System and method for tracking events and providing feedback in a virtual conference

ABSTRACT

A virtual conferencing system is described which tracks events during a virtual conference, provides a user interface for reviewing the events, and allows a moderator and/or other participant to provide feedback related to the events. For example, one embodiment of a virtual conferencing system comprises: a plurality of clients operated by participants and at least one moderator of a virtual conference, each of the clients comprising state management logic to maintain a current state of the virtual conference; a virtual conferencing service to establish audio and/or video connections between the plurality of clients during the virtual conference, the virtual conferencing service further including a state synchronization service communicatively coupled to the state management logic on each client to ensure that the current state of the virtual conference is consistent on each client; a virtual conferencing graphical user interface (GUI) to be rendered on the plurality of clients, the virtual conferencing GUI configured, via signals sent from the state synchronization service, to display a video stream of one or more current speakers during the virtual conference utilizing the established video connections; and a contribution identification module to identify events related to actions of participants during the virtual conference; an event log to store the events identified by the contribution identification module; and an event filter to provide options for searching for specific types of events within the event log based on input from the moderator and/or participants when reviewing a recording of the virtual conference, the event filter to generate a filtered set of events based on the input from the moderator and/or participants.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 14/840,471, filed Aug. 31, 2015, entitled “System And MethodFor Tracking Events And Providing Feedback In A Virtual Conference,”which claims the benefit of and priority to U.S. Provisional PatentApplication No. 62/046,859, filed Sep. 5, 2014, entitled “DecisionSupport System In A Web Video Conferencing Systems,” U.S. ProvisionalPatent Application No. 62/046,879, filed Sep. 5, 2014, entitled“Discussion Support Systems In Web Conferences,” and U.S. ProvisionalPatent Application No. 62/046,880, filed Sep. 5, 2014, entitled“Feedback Systems In Web Conferences,” all of which is hereinincorporated by reference.

BACKGROUND

Field of the Invention

This invention relates generally to the field of computer systems. Moreparticularly, the invention relates to a system and method for trackingevents and providing feedback in a virtual conference.

Description of the Related Art

“Web conferencing” or “virtual conferencing” refers to various forms ofonline collaborative services including web seminars, webcasts, andpeer-level web meetings. Web conferencing systems today supportreal-time audio and video streaming between participants, typicallyunder the coordination of a central web conferencing server.Applications for web conferencing include online classrooms, meetings,training sessions, lectures, and seminars, to name a few.

Participants to a web conference such as students in a virtual classroomcan benefit from formative feedback on their contributions duringclassroom discussions. Such feedback requires identifying, classifying,and/or assessing each contribution (e.g., spoken contribution, writtencontribution) by each individual participant, which may be timeconsuming and impractical. Notwithstanding the pedagogical value offormative feedback and assessment, the time and resources required inproviding such feedback and assessment to participants can prevent ordiminish this learning opportunity for participants.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained from thefollowing detailed description in conjunction with the followingdrawings, in which:

FIG. 1 illustrates an exemplary system architecture on which embodimentsof the invention may be implemented;

FIG. 2 illustrates one embodiment of a graphical user interface (GUI)which includes a center or primary speaker region and a speaker queue;

FIG. 3 illustrates one embodiment of an architecture for selectingbreakout groups;

FIG. 4 illustrates a class initialization graphic for providing aparticipant entry into a virtual classroom;

FIG. 5 illustrates an exemplary pre-class discussion region includingvideo images of participants;

FIG. 6 illustrates a group of students connecting to the virtualclassroom using a shared projector;

FIGS. 7A-B illustrates one embodiment of the graphical user interface asa professor or other moderator joins the virtual classroom;

FIG. 8 illustrates a visual display of materials used by the speaker anda smaller video region for the speaker;

FIG. 9 illustrates a professor or teacher meeting with a student duringoffice hours;

FIG. 10 illustrates visual results of a question on which participantshave voted;

FIG. 11 illustrates one embodiment in which content is displayed on topof the primary speaker region;

FIG. 12 illustrates one embodiment of a graphical user interfacecomprising two current speaker regions;

FIG. 13 illustrates one embodiment of a graphical user interface for atouch screen device;

FIG. 14 illustrates one embodiment of a graphical user interface forinitializing a breakout group;

FIGS. 15A-B illustrate exemplary breakout group materials, annotationsmade to the materials, and video images of participants;

FIGS. 16A-C illustrate an exemplary graphical user interface comprisingmultiple speaker regions, and a region for evaluating studentperformance data when visiting a professor or teacher during officehours;

FIG. 17 illustrates an architecture for synchronizing state betweenclients in accordance with one embodiment of the invention;

FIG. 18 illustrates additional details of the exemplary architecture forsynchronizing state between clients;

FIG. 19 illustrates one embodiment of an architecture for distributingaudio and video among participants;

FIG. 20 illustrates one embodiment of an architecture for storing andreplaying video and audio captured during a class or other virtualconference type;

FIGS. 21A-B illustrate embodiments of a graphical interactive timeline;

FIGS. 22A-B illustrate different logic and processes for generating aninteractive timeline in accordance with one embodiment of the invention;

FIG. 23 illustrates an exemplary lesson plan in a human-readable format;

FIG. 24 illustrates an exemplary interactive timeline and a mappingbetween a YAML representation and the interactive timeline;

FIGS. 25A-C illustrate embodiments of a graphical design interface fordesigning interactive timelines;

FIGS. 26A-B illustrate one embodiment of an architecture for generatinga graphical interactive timeline;

FIG. 27 illustrates how functions associated with a segment of atimeline may be synchronized with clients participating in a virtualconference;

FIGS. 28A-C illustrate an interactive timeline displayed on a secondarydisplay in accordance with one embodiment;

FIG. 29 illustrates a computer system in accordance with one embodimentof the invention.

FIG. 30 illustrates an exemplary decision support module employed in oneembodiment of the invention;

FIG. 31 illustrates an exemplary graphical user interface foridentifying students recommended for participation;

FIG. 32 illustrates a method in accordance with one embodiment of theinvention;

FIG. 33 illustrates an example feedback provision module configured toproviding assessment of participants based on defined learning outcomes;

FIG. 34 illustrates an example event timeline chart;

FIG. 35A illustrates an example event list;

FIG. 35B illustrates an example event list;

FIG. 36 illustrates an example event filter control;

FIG. 37A illustrates an example user interface for selecting a learningoutcome;

FIG. 37B illustrates an example user interface for selecting a learningoutcome;

FIG. 37C illustrates an example user interface for selecting a learningoutcome;

FIG. 38 illustrates an example evaluation rubric;

FIG. 39 illustrates an example user interface for evaluating aparticipant's contribution;

FIG. 40 illustrates an example user interface for evaluating aparticipant's contribution;

FIG. 41 illustrates an example user interface for evaluating aparticipant's contribution;

FIG. 42 illustrates an example user interface for a participant toreview evaluations;

FIG. 43 illustrates an example user interface for a participant toreview evaluations;

FIG. 44 illustrates a flow diagram of an example process of creating andmaintaining a debate in web conferences;

FIG. 45 illustrates an example discussion support module configured tosupport discussion in web conferences;

FIG. 46 illustrates an example user interface for creating a discussion;

FIG. 47 illustrates an example user interface for creating a discussion;

FIG. 48 illustrates an example user interface for creating a discussion;

FIG. 49 illustrates an example user interface for creating a discussion;

FIG. 50 illustrates an example user interface for creating a discussion;

FIG. 51 illustrates an example user interface for evaluating adiscussion participant;

FIG. 52 illustrates an example user interface for evaluating adiscussion participant;

FIG. 53 illustrates an example user interface for enabling or disablingevaluation of discussion participants; and

FIG. 54 illustrates an example user interface for terminating adiscussion.

FIG. 55 illustrates an example user interface with a termination buttonfor terminating a discussion.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the embodiments of the invention described below. Itwill be apparent, however, to one skilled in the art that theembodiments of the invention may be practiced without some of thesespecific details. In other instances, well-known structures and devicesare shown in block diagram form to avoid obscuring the underlyingprinciples of the embodiments of the invention.

One embodiment of the invention described below includes techniques forforming and managing a speaker queue, comprising an ordered set ofparticipants scheduled to speak during the web conference. In oneembodiment, participants may add themselves to the speaker queue byselecting a specified key or set of keys on the keyboard or by selectinga graphic within the graphical user interface. The speaker queue may beimplemented as a first-in-first-out (FIFO) queue in which the currentspeaker is positioned at the first position or beginning of the queueand participants are initially placed at the last position or end of thequeue (i.e., a participant moves towards the top of the speaker queue aseach speaker finishes speaking and is removed from the top of thequeue). In one embodiment, the professor/teacher or other moderator isdesignated to be the “default” speaker, and is always included in adefault position in the queue (e.g., such as the bottom of the queue sothat he/she reaches the top of the queue if no other participants arewaiting to speak).

In addition, one embodiment of the invention includes techniques forsubdividing the participants of a virtual conference into breakoutgroups. One embodiment of the invention automatically generates thebreakout groups based on user profile information associated with eachparticipant, previous interaction history associated with eachparticipant, or any other data associated with participants in thevirtual conferencing environment (e.g., for a group of 20 participants,5 breakout groups of 4 participants each may be formed). One embodimentof the virtual conferencing system initially provides theprofessor/teacher or other moderator a set of recommended breakoutgroups, and then provides the moderator with the ability to edit therecommended groups (e.g., moving users between groups by clicking anddragging user icons via a graphical user interface), either before orafter initiating the breakout groupings. In addition, each breakoutgroup may be automatically provided with a set of materials (includingboth read-only documents and editable shared whiteboards and texteditors) for working on the designated task and for storing the resultsgenerated by each breakout group. Once the breakout group session iscomplete, the virtual conferencing system may display the results foreach breakout group, in turn, and provide each breakout group theopportunity to present the results (e.g., displaying the editedmaterials).

In addition, in some embodiments, an interactive timeline is implementedwhich may be used by a moderator to control the flow of a virtualconference and perform complex operations such as GUI adjustments withminimal effort. In addition, some embodiments provide techniques forautomatically identifying speakers based on specified criteria; storingand filtering events identified during the course of a conference;providing feedback to conference participants (e.g., using the filteredevents); and implementing discussions such as debates between two ormore participants.

A. Architectural Overview

FIG. 1 illustrates a high level system architecture employed in oneembodiment of the invention. In the illustrated embodiment, a pluralityof clients 130, 140, 150, and 160 connect to a virtual conferencingservice 100 over the Internet 180. The clients may comprise any form ofend user devices including desktop/laptop computers (e.g., PCs or Macs),smartphones (e.g., iPhones, Android phones, etc), tablets (e.g., iPads,Galaxy Tablets, etc), and/or wearable devices (e.g., smartwatches suchas the iWatch or Samsung Gear watch). Of course, the underlyingprinciples of the invention are not limited to any particular form ofuser device.

In one embodiment, each of the client devices connect to the virtualconferencing service 100 through a browser or conferencingapp/application 131, 141, 151, 161 which includes a graphical userinterface 132, 142, 152, 162 to allow the end user to interact with thevirtual conferencing service and participate in a virtual conferenceusing the techniques described herein. In addition, each browser/app131, 141, 151, 161 operates in accordance with a current state 135, 145,155, 165 of the virtual conference which is synchronized between theclients 130, 140, 150, 160 using the synchronization techniquesdescribed below. By way of example, and not limitation, the currentstate 135 for client 130 may indicate positioning of the variousgraphical elements within the GUI 132, including the central position ofthe current speaker, a visual indication of the speaker queue, and agraphical representation and/or video images of participants in eachbreakout group.

In the illustrated embodiment, the virtual conferencing service 100includes a persistent state manager 110 for persistently storing updatesto the state of each virtual conference within a state database 115. Asdescribed in detail below, the state may be continually updated inresponse to user input provided via the browsers/apps 131, 141, 151, 161running on the various clients 130, 140, 150, 160. In one embodiment,when a new participant joins the conference, the persistent statemanager 110 provides the client with stored virtual conference statedata required to synchronize the new client state with the state of theother clients participating in the conference. The persistent statemanager 110 may be implemented with a Web server. However, theunderlying principles of the invention are not limited to thisimplementation.

In one embodiment, after the client's state has been initialized fromthe virtual conference state database 115, a dynamic statesynchronization service 120 provides dynamic state updates to the clientin accordance with user input from all of the clients 130, 140, 150, 160during the virtual conference. For example, one embodiment of thedynamic state synchronization service 120 implements a publish/subscribemechanism in which each client publishes its own state updates to thedynamic state synchronization service 120. A client participating in thevirtual conference subscribes to the dynamic state synchronizationservice 120 to receive the published state updates from all otherclients (including itself). Thus, for a virtual conference in whichClients A-D are participants, if Client A publishes a state update(e.g., adding its user to the speaker queue), the dynamic statesynchronization service 120 will forward the update to all subscribingclients (i.e., Clients A-D). This publish/subscribe mechanism isdescribed in greater detail below. In addition, as discussed below,ordering techniques are employed in one embodiment to ensure that thestate updates are applied to each client in the correct order (i.e., toensure that the clients all remain in the same state).

In one embodiment, a multimedia stream distribution service 125 managesthe receipt and distribution of audio and video streams for each of theclients 130, 140, 150, 160. In particular, in one embodiment, eachclient 130, 140, 150, 160 captures audio and/or video of its participantand streams the audio/video to the multimedia stream distributionservice 125, which forwards the audio/video streams to each of theclients 130, 140, 150, 160. The audio is then decoded and output fromspeakers (not shown) and the video is decoded and rendered within eachof the conferencing GUIs 132, 142, 152, 162 (examples provided below).

One embodiment of the multimedia stream distribution service 125 alsoimplements a publish/subscribe mechanism in which each client subscribesto the audio/video streams from every other client. Thus, in the exampleshown in FIG. 1, Client 130 may subscribe to the audio/video streams ofclients 140, 150, and 160. The particular resolution and/or frame rateof each video stream captured on each client may be dependent on thecurrent state 135, 145, 155, 165 of the video conference. For example,when a participant is designated as the current speaker and is providedwith the central speaking position within the GUI, that participant'sclient may capture video having a relatively higher resolution and/orframe rate than when the participant is not the speaker (i.e., and thevideo of the user is rendered within a small thumbnail region of theGUI). Choosing higher quality video for only the current speaker (orcurrent set of speakers if multiple speakers are permitted)significantly reduces the bandwidth requirements of the system.

In one embodiment, a multimedia storage service 190 records audio/videocontent from the virtual conference and other related data to allow themoderator and/or participants to play back and review the virtualconference at a later time. For example, in a classroom environment, aprofessor or teacher may play back portions of the conference to reviewdiscussions among participants or questions which were posed during theconference. The professor or teacher may then provide feedback to theparticipants (e.g., clarifying and issue which was discussed, answeringadditional questions, providing positive reinforcement, etc).

The video and audio content stored on the multimedia storage service 190may be a higher quality than the audio/video used during the livevirtual conference. For example, each individual client may capturehigher quality video and audio than may be possible to stream throughthe multimedia stream distribution service 130. The higher qualityaudio/video may be stored locally on each individual client 130, 140,150, 160 during the virtual conference and may be uploaded to themultimedia storage service 190 following the conference. For example,each time a participant speaks, a local audio clip of the user's voice(e.g., an MP3 or AAC clip) may be recorded and subsequently uploaded tothe multimedia storage service 190. Additionally, state data 135, 145,155, 165 and/or other data required to reconstruct the virtualconference for playback may be stored on the multimedia storage service190 (as described in greater detail below).

The multimedia storage service 190 may be an external service from whichthe virtual conferencing service purchases storage resources. In anotherembodiment, the multimedia storage service 190 is implemented as aseparate module within the virtual conferencing service 100.

Additional details will now be provided for exemplary speaker queue andbreakout group implementations, followed by a description of additionalarchitectural details of the virtual conferencing service 100.

B. Speaker Queue Embodiments

In order to direct the visual attention of conference participantstowards the focus of discussion in a multi-party conversation in avirtual conference, signals sent by participants themselves may berelied on to indicate an intention to speak. In contrast to systemswhich rely solely on speaker volume, this embodiment eliminates possibleerrors due to poor audio conditions, poor network connections, andambiguity in speech patterns. For example, the signals sent by theparticipants can be used instead of or along with speech detectionalgorithms (e.g., manual or automatic).

During a web video conference or virtual conference, meetingparticipants are provided with the ability to request and gain access tothe “center of attention.” For example, as illustrated in FIG. 2, aparticipant has the center of attention if the participant is displayedin the largest video element or in a center element in the virtualconference, referred to as the “current speaker position” 203. In oneembodiment, this is done by a “push-to-talk” or “trigger-to-talk”mechanism where the participant holds down a particular key on thekeyboard, presses a graphical indication in the virtual conferenceenvironment, or performs any other suitable trigger action that wouldindicate that the participant would like to talk, herein referred to asthe “queue key.” The queue key may also toggle the microphone mutestatus if the microphone was previously muted.

By pressing the queue key, the participant places him or herself into aspeaker queue which may be synchronized across all of the clients 130,140, 150, 160 using the dynamic state synchronization service 120 asdescribed herein. As illustrated in FIG. 2, a visual indication 201 ofthe participants in the speaker queue may be rendered within the GUI ofeach client 130, 140, 150, 160. In one embodiment, each client 130, 140,150, 160 maintains its own synchronized copy of the speaker queue. Whena particular participant is added to the speaker queue (e.g., by holdingthe queue key), that participant is automatically added to the localspeaker queue on the participant's client, thereby altering the localstate. The local state change is then synchronized to the other clientsthrough the publish/subscribe mechanism implemented by the dynamic statesynchronization service. If another participant requested entry into thespeaker queue at approximately the same time, the dynamic statesynchronization service 120 resolves the potential conflict using anordering mechanism (described in greater detail below) and propagatescorrect state updates to all of the clients 130, 140, 150, 160.

Thus, by holding the queue key, the participant ensures a place in thespeaker queue and the speaker queue is made visible to all participantsin the virtual conference. In FIG. 2, the visual representation of thespeaker queue 201 displays each participant in the queue throughscreenshots of the video stream of the participant in the virtualconference or any other suitable digital representation of theparticipant (e.g., a picture, avatar, etc). Video of the speaker at thefront of the queue is displayed within the current speaker position 203of the GUI. In addition, in FIG. 2, thumbnails of all other participants202 (or a subset thereof) in the virtual conference are displayed withinthe GUI.

One embodiment of the system tracks how long each participant is in thespeaker queue, how long each participant is given the center ofattention and how much each participant has talked (e.g., based onsignal processing of the participant's visual cue while the participantwas given the center of attention). In one embodiment, this isaccomplished by setting/resetting programmable timers on each of theclients 130, 140, 150, 160 and/or on the virtual conferencing service100. In one embodiment, the time allocated to speak may be controlled bythe professor or teacher (or other moderator).

The same queue key can also be used to control the mute status of themicrophone. If the microphone was previously muted, entering into thespeaker queue by holding the queue key will also un-mute the microphoneallowing the audio of that participant to be heard by all participantsin the virtual conference. In another embodiment, the previously mutedmicrophone may not be un-muted automatically and, instead, themicrophone's status is presented to the participant or all participants.For example, if the microphone was muted prior to pressing the same key(or providing any of the other actions), then pressing the same keypresents an indication that the microphone is currently muted.

The action of the participant joining the speaker queue is communicatedto all other participants via a message or indication such as a speakerqueue visualization or a display of the speaker queue 201. In oneembodiment, this is delivered to clients through the publish/subscribemechanism employed by the dynamic state synchronization service 120.

In one embodiment, one of the participants or a moderator/instructor isset as a “default” speaker (e.g., the professor, leader, or designatedparticipant or student of the participants in the virtual conference)who may be configured as being “behind” the last participant in thespeaker queue. Thus, when the speaker queue is empty, the defaultspeaker is placed in the center and may indicate which participantshould be given the center of attention. The default speaker can bedesignated, for example, by a professor to a student allowing thestudent to field or answer questions after a presentation is given(e.g., by the student).

The speaker queue 201 may be implemented as a First In, First Out (FIFO)queue and may have a default speaker associated with the queue. Forexample, the default speaker would be placed in the last or trailerposition of the speaker queue. In one embodiment, a participant is addedto the speaker queue (e.g., at the end of the speaker queue visually) byselecting a queue key and the participant is kept in the speaker queueby holding the queue key. The queue key can be a control key or anyother suitable key on their keyboard and/or may be implemented as agraphical icon in the GUI (which the user selects via a mouse or atouch-pad). In one embodiment, a participant is removed from the speakerqueue when he or she releases the designated queue key or deselects thegraphical icon.

In one embodiment, the participant at the head of the speaker queue isgiven the center of attention by being visually featured in theconference. For example, the participant's visual cue is placed in acenter element of the virtual conference or placed in the largestelement in the virtual conference (e.g., center speaker position 203 inFIG. 2). Once the participant has been given the center of attention,the participant may be excluded/removed from the displayed speaker queue201.

In one embodiment, the speaker queue is made visible to everyparticipant in the virtual conference in a displayed speaker queue orqueue visualization. For example, the displayed speaker queue 201 may bean array (e.g., horizontal, vertical, curved, etc.) of small photographsor visual cues of participants in the speaker queue. The displayedspeaker queue can be in the bottom left-hand corner of the userinterface of the virtual conferencing environment and positioned fromleft-to-right based on index or position of the participant in thespeaker queue. Of course, the underlying principles of the invention arenot limited to any particular visual representation of the speakerqueue.

When the speaker queue is empty, the default speaker (e.g., in thetrailer position of the speaker queue) is featured in the conference,for example, by being given the center of attention. The leader, webconference initiator, or professor can initially be the default speakerand/or can designate a default speaker. For example, the professor candesignate the default speaker by selecting the designated participant'sthumbnail video feed 202 or other visual cue in the list or group ofvisual cues (e.g., at top, bottom, or side of the virtual conference).In one embodiment, each participant's audio broadcasting is muted bydefault and may be unmuted in response to input from the participant(e.g., by the participant holding the queue key).

In one embodiment, when a participant presses and holds down the queuekey, his or her microphone is un-muted. When the participant releasesthe queue key, the participant's microphone is muted again. In oneembodiment, each speaker queue modification is synchronized to theclients of all participants via the publish/subscribe techniquesimplemented by the dynamic state synchronization service 120. Inaddition, data related to participation in the speaker queue may bestored by the virtual conferencing service 100 (and/or the externalmultimedia storage service 190) and later used to analyze participationactivity (e.g., a length of time each participant was speaking).

While the embodiment in FIG. 2 illustrates a single speaker in a currentspeaker position 203, other embodiments may include multiple currentspeaker positions. For example, one embodiment of the invention includesmulti-region layouts in which the center of attention is sub-dividedinto multiple “attention regions,” allowing for the simultaneous displayof more than one participant in the virtual conference. For example,FIG. 12 (discussed below) illustrates an embodiment with two attentionregions for two different speakers. Another embodiment includes fourattention regions arranged in a square or rectangle with two regionstowards the top of the display and two regions towards the bottom of thedisplay. Any number of attention regions may be generated while stillcomplying with the underlying principles of the invention.

In these embodiments, a single speaker queue may be maintained for allattention regions. When a region becomes available (using the samecriteria as with the single-region center of attention embodimentsdescribed herein), the first participant in the speaker queue is removedand the participant video is displayed in that attention region. In analternate embodiment, each attention region may be assigned its owndedicated speaker queue (e.g., N speaker queues for N attentionregions). This embodiment may be used, for example, to provide adedicated attention region for each breakout group, to allow differentmembers of the breakout group to take turns speaking within eachdedicated attention region. In either of these embodiments, a “defaultspeaker” may also be specified for each attention region.

In addition, in one embodiment, when a speaker occupies an attentionregion in the center of attention, the professor, leader, or designatedparticipant can “pin” the speaker to that region (e.g., by selecting akey or graphical element within the GUI). Pinning a speaker has the sameeffect as if the speaker actively maintained the position by holding thepush-to-talk activation key or alternative mechanism to maintain thefeatured position. In one embodiment, no other speaker will be movedfrom the speaker queue into the speaker position until the featuredspeaker is “unpinned” by the professor, leader, designated participant,or the featured speaker themselves.

C. Breakout Group Embodiments

In a traditional classroom environment or meeting, an instructor ormeeting organizer determines how to subdivide a group (e.g., by havingparticipants count off, dividing into pre-arranged groups or using someother heuristic). Once the groups are organized, the groups typicallyshuffle around the room to a designated spot to work together. Theorganizer may walk around to interact with each group. Oncere-assembled, the groups may take turns presenting.

One embodiment of the invention provides support for the samefunctionality within a virtualized conferencing environment. Breakoutgroups can be formed by the virtual conferencing environment based onuser profile information associated with each participant, previousinteraction history associated with each participant or any othersuitable historical data associated with each participant in the virtualconferencing environment. For example, this information includes pastparticipation statistics associated with the participant, grades,performance in assignments, etc.

In another embodiment, the participant leading the virtual conferencecan also affect how the breakout groups are formed. For example, theparticipant can select to move participants between the formed breakoutgroups (e.g., using a graphical click-and-drag mechanism or othersuitable actions), or indicate which participants should be in the samebreakout group when the breakout groups are formed.

The participant leading the virtual conference can also determine astart and/or an end time associated with the session of formed breakoutgroups, for example, indicating when the breakout groups are formed andwhen the breakout groups are dissolved into additional breakout groupsor one big group.

In one embodiment, each breakout group is provided with a copy of allassociated materials and/or resources from the main group (e.g., aclass) and can include any additional materials and/or resources neededto perform an assigned task or other suitable action in the virtualconference. Any participant may be provided with the ability to uploadany type of material, as appropriate. Furthermore when the breakoutgroups are re-assembled into one big group or one or more additionalbreakout groups, the participant leading the virtual conference canaccess and feature the participants and their work (e.g., through thematerials and/or additional materials).

One embodiment of a logical architecture and flow for forming breakoutgroups is illustrated in FIG. 3. This architecture may be implemented insoftware modules executed within the virtual conferencing service 100,on the client machines 130, 140, 150, 160, or any combination thereof(e.g., with some operations performed on the virtual conferencingservice and some on the clients).

In one embodiment, an active conference 310 is formed as participantslog in and authenticate with the virtual conferencing service 100 (e.g.,as participants arrive for class). A user database 305 containing userIDs and other pertinent information may be queried during the loginprocess to uniquely identify each user. In one embodiment, a breakoutgroup selection module 320 selects participants to be subdivided intobreakout groups in accordance with input from the moderator 325 (e.g., aprocessor or instructor), the identity of active participants in theconference 341, and other user data 306 which may be retrieved from theuser database 305 (or a different database).

By way of example, and not limitation, the moderator input 325 mayindicate that the moderator wishes for there to be four breakout groups,with randomly selected participants. In response, the breakout groupselection module 320 will subdivide the active participants 341 intofour groups, as close in size as possible. For example, if there are 28students, then four groups of 7 participants will be created. If thereare 26 students, then two groups of 7 and two groups of 6 will beformed. Rather than randomly selecting the participants, the breakoutgroup selection module 320 may run through the list of activeparticipants alphabetically (e.g., using the first or last names of theparticipants).

Alternatively, the participants in each breakout group may bepre-assigned by moderator ahead of the class or other meeting. In thisembodiment, all that is required by the breakout group selection module320 is the list of active participants 341.

In one embodiment, the breakout group selection module 320 may select aninitial set of breakout groups which the moderator may then review andmodify. For example, the initial set may be selected based on userprofile data or other pertinent data 306 stored in the user database 305such as the performance of each user in the class (e.g., ensuring thateach group includes at least some higher performing participants).Performance may be based on the current grade of each participant in theclass, the cumulative time that each participant has talked, the gradeon a recent exam, and/or additional information provided by themoderator.

The breakout group selection module 320 may consider other pertinentdata to generate the initial set of breakout groups such as participantrelationships (e.g., frequency of interactions among participants);specific exercise outcomes; results from a poll (e.g., automaticallygrouping together participants who had similar or different responses);differing responses (e.g., automatically grouping together participantswho had differing responses, in order to maximize likelihood of aproductive learning exchange among participants); pre-class work; andorder of arrival time to virtual conference or presence in virtualconference, to name a few. In one embodiment, the moderator may alsospecify a maximum size for breakout groups. The breakout group selectionmodule 320 will then form the breakout groups in accordance with thismaximum size.

In one embodiment, breakout groups may be formed by an indication or atrigger from a participant or moderator (e.g., selection of a button,voice activated). The indication or trigger may be implemented withinthe virtual conference GUI or may be specified on a second screen ormobile device connected to the virtual conference.

In one embodiment, once a breakout group is formed, the members of thebreakout group will only receive and render video and/or audio of othermembers of the breakout group. The video/audio of the moderator may alsobe shared with the members of a breakout group when visiting thebreakout group. This may be accomplished, for example, by muting theaudio and disabling video rendering of streams for participants in allother groups. in another embodiment, the publish/subscribe mechanism inthe multimedia stream distribution service 125 is updated to that aclient only subscribes to the audio/video stream of other participantsin the group. Various other mechanisms may be employed to ensure thataudio is contained within each breakout group.

In one embodiment, End-of-Breakout indications are generated, warningwhen breakout groups are about to end and/or that the breakout groupswill be formed into additional breakout groups or a larger group (e.g.,the original group). The indications maybe visual (e.g., via a pop-upwindow), audible (e.g., via an alarm or ringtone), or any combinationthereof.

In addition to having the ability to “visit” breakout groups, theprocessor or teacher may broadcast audio/video or messages to all of thebreakout groups, and may also receive messages from one or more of thebreakout groups (e.g., questions posed by participants).

Returning to FIG. 3, once the breakout groups 328 have been selected(i.e., the users in each breakout group identified using the techniquesdescribed above), breakout group generation logic 330 generates aninstance of each breakout group, which may include (among other things)copies of materials from a materials database 334 to be used during thebreakout sessions. In FIG. 3, for example, group materials 351 and 352are provided to breakout groups 341 and 342, respectively. In oneembodiment, the group materials 351 and 352 are copies of materials fromthe materials database 334 which may be independently edited by eachrespective group, 341 and 342. In addition, the same set of sharedmaterials 360 may also be made available and shared by each breakoutgroup.

In one embodiment, the materials and/or resources that may bedistributed to all breakout groups include (but are not limited to)YouTube videos; PDF files; PowerPoint files; URLs; document notes;picture files in different forms; sound files (e.g., MP3); links toonline sites; and any other visible or audible material capable of beingreviewed and/or edited during for the breakout session.

In one embodiment, each the participants in a breakout group areprovided with a shared text editor and whiteboard function via a noteelement in the virtual conference. The shared text editor may beimplemented by program code executed on each client and/or the virtualconferencing service 100. Each participant in a breakout group can alsoadd material or documents not visible to other breakout groups. Theseadditional external materials may be kept private to the participants ofthe specific breakout group (i.e., stored as group materials 351-352 inFIG. 3).

In one embodiment, each breakout group is provided with a tool to drawand annotate on top of shared or private materials or resources. Theannotation tool may be provided as program code executed on each client130, 140, 150, 160 or on the virtual conferencing service 100 (or both).

One embodiment of the invention provides for group-specific dispersionof material. For example, the professor, teacher or other form ofmoderator may send particular documents and other resources (e.g.,homework) to specific breakout groups (e.g., based on participants inthe breakout group).

As mentioned, in one embodiment, the moderator (e.g., professor orteacher) send a written-text or spoken-audio message to all breakoutgroups and may join a breakout group and leave the breakout group toreturn to a bird's-eye overview of all breakout groups. In addition, themoderator may audibly listen to all/each breakout group individuallywithout joining each breakout group and may oversee work happeningwithin all breakout groups. The moderator may also view the materialsbeing edited by each of the breakout groups (e.g., shared notes as theyare being typed; whiteboards, as they are being drawn, annotations asthey are being added). The moderator may further respond to individualquestions raised by specific breakout groups; move/drag a participantfrom one breakout group into another breakout group or out of the formedbreakout groups completely; and cancel breakout group formation andreturn to additional breakout groups or one big group.

In one embodiment, a breakout group can be featured (to otherparticipants not in the breakout group). For example, the moderator mayselect the breakout group (e.g., click, voice activate) resulting in thepresentation of the breakout group (and all the participants in theselected breakout group) in the center of the virtual conference. In oneembodiment, when a breakout group is presenting, the dynamic statesynchronization service 120 will ensure that the state updates on eachclient cause the members of the breakout group to have the center ofattention. The moderator may also minimize the presentation of otherparticipants not in the selected breakout group. Materials associatedwith the selected or featured breakout group may be presented in asimilar manner.

Additional graphical user interface (GUI) features are illustrated inFIGS. 4-18. FIG. 4 illustrates an embodiment which may be displayed oncea participant has logged in to the virtual conferencing service. A classinitialization graphic 401 provides an indication of the amount of timebefore class begins (5 minutes in the example). The user may select thegraphic to enter into the virtual classroom.

Once the participant has selected the class initialization graphic 401,the participant is taken to a pre-class user interface such as shown inFIG. 5. In this embodiment, video thumbnails of other participants whohave logged in to the classroom are displayed within a pre-classdiscussion region 501. A set of tools 502 are also provided to allowusers to text one another, open personal video chat sessions, etc.

FIG. 6 illustrates multiple students sharing a single screen on aprojector for the class and using separate devices (e.g., computers,tablets, etc) to interact with the virtual classroom (e.g., to speak inthe center position, provide text comments and/or questions, etc).

FIG. 7A illustrates the graphical user interface when the professorinitially joins the video conference. As illustrated, the participantthumbnails 701 are arranged randomly around the main display. Incontrast, in FIG. 7B, the participant thumbnails 701 have been moved inan organized manner to the top of the display (no longer obscuring theprimary speaker region). The order of the participant thumbnails 701 maybe alphabetical or based on other variables such as grades or classperformance.

As mentioned, the current speaker may rely on various visual materialsduring the class such as a PowerPoint presentation or other graphicalmaterials. FIG. 8 illustrates one embodiment in which the speaker isrelying on materials 802 which are displayed in the center region of thedisplay. In one embodiment, this is enabled by providing the speakerwith a control button (physical or graphical) which, when selected,allows the speaker to identify materials to be displayed during theclass. The video image of the speaker is offset to a thumbnail image 801towards the bottom of the display which is differentiated from theparticipant thumbnails 701 based on location and size (i.e., the speakerthumbnail 801 is larger than the participant thumbnails 701).

In one embodiment, the professor uses gesture controls to manipulate thecontent in the speaker materials. For example, in FIG. 8, the professoris rotating his hand to cause the image of a human brain to rotatewithin the primary display region. Gesture controls may be implementedby capturing sensor data from a motion controller attached to theprofessor's computer, and using it to modify or reposition (e.g. 3-Drotate) the content. Through the publish-subscribe mechanism, the streamof sensor data that triggered these modifications can be replicated inthe view of all other clients in the class/conference.

In one embodiment, students/participants are provided with a graphic to“raise a hand” during the class/conference. The professor or othermoderator will be provided with a visual indication of a student raisinga hand (e.g., via the student's thumbnail being highlighted with a handicon or other form of highlight graphic) and may acknowledge the studentby selecting the student's thumbnail. FIG. 9 illustrates a video region901 for displaying the video feed of the student who has beenacknowledged by the professor. She is brought into the main element orcenter element along with the professor in response to the professor'sacknowledgement.

FIG. 10 illustrates a poll which is conducted for the purpose of formingbreakout groups. That is, the breakout groups are initially determinedby participants' answers to the poll. The breakout groups can includethe participants who voted similarly or can be a mixed group includingparticipants who all voted differently. In the illustrated embodiment,the participants' answers are shown but in another embodiment theanswers can be anonymous.

FIG. 11 illustrates a GUI feature in which the professor has selected anoption to overlay material over the primary speaker region in the centerof the display (in contrast to the embodiment shown in FIG. 8 in whichthe material is displayed in the center and the video image of theprofessor is offset to the side). The professor may specify this optionusing a different control key or graphic. The overlaid material in thisembodiment may also be a real-time simulation.

FIG. 12 illustrates an embodiment which includes two primary speakerregions 1201-1202 within the GUI. This embodiment may be used, forexample, to enable debates between two or more participants or to allowtwo representatives of a breakout group to present results. Additionalusers may be added within additional speaker regions. For example, Nadjacent regions may be used for N different users during a debate orduring a breakout group presentation. In one embodiment, the thumbnailsof the users may be removed from the participant thumbnail region 701when the participants are shown in the current speaker regions1201-1202.

As mentioned, in one embodiment, users are provided with the ability toview and annotate material via a touch-screen device such as a tabletdevice. FIG. 13 illustrates one embodiment of a tablet on which materialis presented and annotations of material are made by a participant ofthe conference. A participant (e.g., as shown in the slightly enlargedvisual cue on the top of the GUI) presents material and can annotate thematerial in front of the class or conference. Each participant may ormay not have the ability to annotate the material. In one, the professoris provided with the ability to annotate the material and may grantaccess to other participants.

FIG. 14 illustrates an exemplary message and GUI displayed toparticipants who are about to be placed in a breakout group. In theillustrated embodiment, the GUI includes a set of breakout groupthumbnails comprising still pictures or video of the participants in thebreakout group.

FIGS. 15A-B illustrates an exemplary set of breakout group GUI includinga vertically arranged set of breakout group thumbnails 1501, breakoutgroup materials 1502, and notes 1503 recorded by the breakout group. Inaddition, FIG. 15B shows how the breakout group materials 1502 may beedited with annotations 1504 (e.g., performed via a touchscreen, mouse,or other user input device).

In one embodiment of the invention, the professor or teacher may beavailable to meet with students during office hours. FIG. 16Aillustrates an exemplary embodiment in which video of a participant isdisplayed in current speaker region 1601 as the participant is meetingwith a professor during office hours, with video of the professordisplayed in current speaker region 1602. FIG. 16B illustrates anexemplary embodiment in which the student and professor review thestudent's performance in the class, as indicated by student performancedata 1605. In this embodiment, video of the student and professor isdisplayed within thumbnail images 1605. As illustrated in FIG. 16C, thestudent and professor may review the student's participation during theclass, which is replayed in region 1610. As previously discussed, theaudio and/or video from the class may be stored and replayed from theexternal multimedia storage service 190.

D. Additional Architectural Embodiments

As mentioned above, in one embodiment of the invention, the dynamicstate synchronization service 120 interacts with the various clients130, 140, 150, 160 to ensure that the state of each client is consistent(e.g., the current state of the speaker queue, the identity of theparticipant currently in the center speaker position, the identity ofparticipants in each breakout group, etc). As illustrated in FIG. 17,one embodiment of the dynamic state synchronization service 120 includespublish-subscribe logic 1721 which allows each client to subscribe toreceive state updates for every other client. In one embodiment, thepublish-subscribe logic 1721 maintains a publication queue for eachclient and every client subscribes to the publication queues of everyother client (i.e., to ensure that all state updates are received byevery client). Thus, when client 130 transmits a state update to itspublication queue, all of the clients 130, 140, 150 which subscribe toclient 130's publication queue receive the state update.

In addition, in one embodiment, sequence numbering logic 1722 ensuresthat state updates are applied to each client in the correct order. Forexample, the sequence numbering logic 1722 may increment a counter inresponse to the receipt of each new state update received from eachclient. The current counter value may then be attached to each stateupdate to ensure that the state updates are applied in the order inwhich they are received by the dynamic state synchronization service120. For example, the publish-subscribe logic 1721 may construct apacket for each state update and may embed the counter value within afield in each packet prior to transmission to each client 130, 140, 150.

In one embodiment, each client 130, 140, 150 includes state managementlogic 1701, 1702, 1703, respectively, which processes the state updatesto maintain a consistent local state 135, 145, 155, respectively. Thestate management logic 1701, 1702, 1703 maintains a global reorderbuffer 1711, 1721, 1731 into which all of the state updates areinitially stored. Because packets may sometimes be received over theInternet out of order, the global reorder buffer is used to reorder thestate updates when necessary to ensure that the state updates areapplied in the same order as the counter values associated with eachstate update.

In addition, in one embodiment, the state management logic 1711, 1721,1731 assigns a publisher sequence number to indicate the order of stateupdate generated locally on its client 130, 140, 150, respectively. Forexample, if a participant on client 130 generates a request to be thecurrent speaker, then sends a request to ask a question, and thenremoves the request to be the current speaker, the state managementlogic 1701 may assign a sequence number to each of these state updatesto indicate the order in which they were submitted. The publishersequence numbers are transmitted along with each state update to thepublish-subscribe logic 1721 and are received by each individual client.To ensure that the state updates are applied in the same order as theywere generated, the state management logic 170, 1702, 1703, maintains aset of publisher reorder buffers 1712-1714, 1722-1724, 1732-1734,respectively, which may be chained to the global reorder buffers 1711,1721, 1731, respectively. The state management logic 1701-1703 reordersthe state updates within the publisher reorder buffers 1712-1714,1722-1724, 1732-1734 in accordance with the publisher sequence numbersto ensure that the state updates are applied in the same order in whichthey were generated on each client.

The end result is that the global order of state updates is maintained,based on the order in which state updates are received by thepublish-subscribe logic 1721 and program order is maintained based onthe sequence of operations performed on each individual client.

Because participants may arrive to the virtual classroom (or other typeof virtual conference) at different times, one embodiment of theinvention includes techniques for initializing each newly-arrived clientwith the correct state. As illustrated in FIG. 18, this is accomplishedin one embodiment with the persistent state manager 110 (brieflymentioned above) which maintains the current state of each client withina state database 115. Each time a state update is generated at a client,that client initially transmits an indication of the state update to thepersistent state manager 110, which stores the state update within thestate database 115. The client then connects with the publish-subscribelogic 1721 to publish the state update to the other clients. Thus, thestate database 115 contains a persistent representation of the currentstate of all of the clients.

In one embodiment, when a new client 1810 comes online (e.g., inresponse to the participant joining an ongoing class), its statemanagement logic 1820 performs the following operations to initializeits local state 1815. In one embodiment, the state management logic 1820first establishes a connection with the publish-subscribe logic 1721,subscribing to all state updates published by all other clients and toits own state updates (as previously described). It then beginsbuffering all state updates received from the publish-subscribe logic1721. In one embodiment, the state management logic 1820 then connectswith the persistent state manager 110 to receive a copy of the currentpersistent state stored in the state database 115. Given transactionaldelays over the Internet, during the period of time when the initialconnection is made to the persistent state manager 110 and the time whenthe state is downloaded from the state database 115, there may bechanges made to the persistent state within the state database 115.Moreover, some state updates which the state management logic 1820receives from the publish-subscribe logic 1721 may already be reflectedin the state database 115 (i.e., because the state management logic 1820connects first to the publish-subscribe logic 1721). Consequently,following the retrieval of the state from the state database 115 thestate management logic 1820 may have a superset of all of the state dataneeded to initialize its local state 1815. It may include redundantstate updates—some of which are reflected in the persistent state fromthe state database and some of which were received from thepublish-subscribe logic.

To ensure that these redundancies are resolved consistently, oneembodiment of the invention ensures that all state updates areidempotent. As understood by those of skill in the art, idempotence is aproperty of operations in computer science that can be applied multipletimes without changing the result beyond the initial application. Thus,for example, if the participant on client 130 requests to be added tothe speaker queue, this state update may be applied multiple times onthe new client 1810 (e.g., once from the state database 115 and oncefrom the publish-subscribe logic 1721) to achieve the same local state1815 (i.e., the second application of the state update will not alterthe final local state 1815). Thus, by ensuring that all state updatesare idempotent, redundant state updates may simply be applied multipletimes without affecting the underlying local state of each client.

In summary, once the state management logic 1820 has received andapplied the copy of the persistent state from the state database 115 andapplied all of the state updates received from the publish-subscribelogic (some of which may be redundant), the local state 1815 on the newclient 1810 will be consistent with the local states 135, 145 of theother clients 130, 140.

In order to ensure a responsive user interface, one embodiment of thestate management logic 1820 applies speculative state updates locally,in response to input from the local participant, and then resolves thestate updates to ensure consistency upon receipt of state updateresponses from the publish-subscribe logic 1721. For example, inresponse to the participant on client 1810 selecting and holding thequeue key, the state management logic 1820 may instantly place theparticipant in the speaker queue and/or place the participant in thecenter speaker region (if the participant is first in the queue). Thus,the state update will be instantly reflected in the graphical userinterface of the participant, resulting in a positive user experience.

The state management logic 1820 then transmits the state update to thepublish-subscribe logic 1721 where it is assigned a sequence number asdescribed above. Because the client 1810 subscribes to its ownpublication queue as well as those of all other clients, the client 1810will receive the state update from the publish-subscribe logic 1721.Upon receiving its own state update, both the global and publisherreorder buffers are applied to ensure proper ordering, and then theupdate is re-applied to client 1810. The second application of the stateupdate ensures state consistency since the proper ordering ismaintained. Re-applying an update is safe to do because of the idempotent property of state updates, as mentioned above.

There is the possibility of flicker in the user interface if there wasan intervening, conflicting update to client 1810 between the firstapplication of the state update and the second. That flicker will notaffect state consistency, but it can cause a visual effect that isundesirable to the user. In one embodiment, some instances of flickerare eliminated by explicitly detecting conflicting state updates. Todetect conflicting state updates, each incoming state update to client1810 is checked against a queue of speculatively applied state changesto see if it will affect state that was speculatively applied. If aconflicting incoming state update is detected, client 1810 will notapply that update in one important case, specifically when client 1810has already applied the state update as a speculative update (i.e.,client 1810 published the state update) and no other conflicting stateupdates have been detected. This optimization eliminates flicker when,for instance, a user requests entry into the speaker queue and thenquickly (in less than the round trip time to the publish-subscribeserver) requests to be removed from the speaker queue.

As illustrated in FIG. 19, in one embodiment, the multimedia streamdistribution service 125 includes stream forwarding logic 1820 formanaging the receipt and distribution of audio and video streams foreach of the clients 130, 140, 150. In particular, in one embodiment,each client 130, 140, 150 captures audio and/or video of its participantand streams the audio/video to the stream forwarding logic 1920, whichforwards the audio/video streams to each of the clients 130, 140, 150. Avideo camera 1910 and microphone are illustrated for capturing video andaudio of the participant, respectively. Each client 130 also includes adisplay on which the GUI 132 is rendered and speakers 1912 forgenerating the audio of other participants. In one embodiment,audio/video (NV) compression and decompression logic 1902 compressesaudio and video of the participant and the compressed audio/video isthen streamed to the stream forwarding logic 1920 by the virtualconferencing app or browser 1901. While the A/Vcompression/decompression logic is shown integrated within theapp/browser in FIG. 19, this may be a logically separate module which isaccessed by the app/browser.

In one embodiment, the app/browser 1901 of each client 130, 140, 150establishes a web socket connection with the stream forwarding logic1920 to receive streams from each of the other clients. The streamforwarding logic 1920 may distribute audio/video using apublish/subscribe mechanism where each client subscribes to the audioand video feeds of all other clients. The stream forwarding logic thenforwards the incoming audio/video feeds to all subscribing clients.

Upon receiving the audio and video from other clients, the A/Vdecompression logic 1902 decompresses/decodes the audio and videostreams, renders the video within the GUI (e.g., within the thumbnailimages or within the center speaker region as described above) andoutputs the decoded audio through the speakers 1912.

In one embodiment, the A/V compression/decompression logic 1902 adjuststhe compression on the video of the participant depending on the size ofthe video image of the participant shown within the GUI. For example, ifthe participant is the current speaker (i.e., at the top of the speakerqueue), the A/V compression/decompression logic 1902 may encode thevideo at a relatively higher resolution and/or frame rate, because ahigher resolution is needed to provide an acceptable level of videoquality for the relatively larger speaker region. In contrast, if theparticipant is not the current speaker, then thecompression/decompression logic 1902 may encode the video at arelatively lower resolution and/or frame rate to provide an acceptablequality for displaying video within a thumbnail region. The app orbrowser 1901 may determine the required size of the video image (e.g.,whether the user is the current speaker) by reading the local state data135 stored on the client. In one embodiment, the app/browser 1901 mayspecify a desired bitrate to the A/V compression/decompression logic1902 which will then adjust the resolution and/or frame rateaccordingly. These techniques will help to keep the bitrate at areasonable level because if there is only one speaker, for example, thenonly one high quality stream will be transmitted and sent to allclients. In one embodiment, when a new participant becomes the currentspeaker, this will be reflected in the state data of each client and theapp or browser will control the A/V compression/decompression logicaccordingly (i.e., to increase the resolution and frame rate of thevideo stream showing the new speaker).

In one embodiment of the invention, each app or browser 1901 performsdynamic bitrate adaptation based on the bitrate available to each clientand the requirements of the various video regions within the GUI. Forexample, if 2 Mbps is available to a particular client 130, then (usingFIG. 12 as an example GUI) the app/browser 1901 may specify to the A/Vcompression/decompression logic 1902 to allocate 1 Mbps to encode bothof the current speaker regions 1201-1202 and may allocate the remaining1 Mbps to encode all of the participant thumbnails 701. The A/Vcompression/decompression logic 1902 will then compress/decompress videoin accordance with the allocated bitrates for each region of the GUI. Inaddition, in one embodiment, each participant may be provided theability to select different quality levels to be used when encoding theparticipant's outgoing video stream. By way of example, these selectablelevels may include high quality, low quality, and audio only (i.e., novideo feed).

As mentioned, the multimedia storage service 190 may capture and storeaudio and video of a class (or other virtual conference) for subsequentplayback. As illustrated in FIG. 20, in one embodiment, the multimediastorage service 190 may be treated like any other client and may beconfigured to receive and record all audio/video streams for allparticipants on a storage device 2000. The data format used may comprisea plurality of audio and video clips of each of the participants. Inaddition, a timestamp may be associated with each audio and video clipwhich may be used to reconstruct the playback of the virtual class(i.e., to ensure that each audio and video clip is played back at theappropriate time).

As mentioned above, the video and audio content stored on the multimediastorage service 190 may be a higher quality than the audio/video usedduring the live virtual conference. For example, as illustrated in FIG.20, local audio and/or video capture logic 2005 on each individualclient may capture higher quality video and audio than may be possibleto stream through the multimedia stream distribution service 130. Thehigher quality audio/video may be stored locally, as a set of audioand/or video clips on a storage device 2001 of each client 130 duringthe virtual conference. When the conference has ended, these clips maybe uploaded to the storage device 2000 on the multimedia storage service190. For example, each time a participant speaks, a local audio clip ofthe user's voice (e.g., an MP3 or AAC clip) may be recorded andsubsequently uploaded to the multimedia storage service 190.Additionally, state data 135, 145, 155, 165, timestamp data, and/or anyother data usable to reconstruct the virtual conference for playback maybe collected and stored on the multimedia storage service 190 (asdescribed in greater detail below).

In one embodiment, the recorded audio/video from the virtual conference2000 may include audio/video and other content generated by each of thebreakout groups. In this embodiment, each of the audio/video clips maybe associated with an identifier identifying the breakout group fromwhich they were collected. In this manner, the professor or teacher mayindividually play back the audio/video and other content to reconstructand review the discussion and content generated by each breakout group.

In one embodiment, playback of audio, video, and other content isperformed using a virtual conference playback tool. The playback toolmay be implemented as a separate app or application or as a browserplug-in.

While the embodiment described above relies on a central virtualconferencing service 100 to establish connections between clients and tostream video and audio between the clients, the underlying principles ofthe invention are not limited to this particular implementation. Forexample, in one embodiment, the clients are configured to establishpeer-to-peer connections with one another, either without a centralserver (e.g., using a peer-to-peer networking protocol), or using acentral server solely as a directory server, to lookup the networkaddresses of the other clients participating in the virtual conference.Once connected via peer-to-peer connections, the clients may implementthe same state synchronization techniques described above, includingmanagement of the speaker queue and breakout groups. In addition, inthis embodiment, the clients establish direct connections with oneanother to exchange video and audio of the participants.

Alternatively, rather than merely forwarding video and audio streamsbetween participants, the central virtual conferencing service 100 maycompress/recompress the video and/or audio based on the capabilities ofeach individual client (e.g., reducing the resolution and/or frame ratefor clients with smaller displays or lower-bitrate network connections).In addition, in one embodiment, the virtual conferencing service 100 maycombine the video streams from each of the clients into a single videostream that is then streamed to all of the clients (e.g., compositingall of the video streams into a single video frame, which is thencompressed and streamed to the clients).

In addition, various different forms of video and audio compression maybe used by the clients and/or the virtual conferencing service 100 whilestill complying with the underlying principles of the invention. Thisincludes, but is not limited to, H.264, VP8, and VP9 for video codingand Opus and iSAC for audio coding.

System and Method for a Virtual Conference Interactive Timeline

As mentioned above, in some virtual conferencing systems, the meetingorganizer or moderator is provided with control over the state of thevirtual conferencing system via a graphical control panel. For example,when it is time to set up a debate between two or more students, theprofessor uses the control panel to manually rearrange the graphicaluser interface to include two or more speaker positions and identifiesthe students to participate in the debate. Similarly, to subdivide theclass into breakout groups, the professor uses the control panel tomanually specify the size of the breakout groups, identify the studentsin each group, provide the necessary materials for each group to useduring the breakout session, and specify the duration of the breakoutperiod. When the breakout period is over, the professor again uses thecontrol panel to rearrange the graphical user interface to review theresults of each breakout group. As another example, when a poll is to beconducted, the professor uses the control panel to initiate the poll,which may involve additional modifications to the graphical userinterface.

Requiring the instructor (or other moderator) to manually perform all ofthe above operations during the course of a class (or other type ofvirtual conference) can be distracting and time consuming. To addressthis problem, one embodiment of the invention comprises an interactivevideo conferencing timeline which includes a graphical representation ofeach of the ordered events scheduled to occur during the course of avirtual conference. To perform the sequence of operations required toimplement an event, the professor (or other moderator) simply selectsthe graphical representation corresponding to the event. In an alternateimplementation, the graphical representations may be selectedautomatically by the system, in accordance with timing informationassociated with each event.

While the remainder of the discussion below will focus on an onlineclassroom implementation, the underlying principles of the invention maybe implemented in any virtual conferencing environment in whichdifferent events require changes to the virtual conferencing systemconfiguration.

FIG. 21A illustrates an exemplary graphical interactive timeline 2150for use in an online classroom in which the lesson plan for the class issubdivided into a plurality of “sections” 2110-2111 and each section issubdivided into a plurality of “segments” 2120-2123, 2130 correspondingto scheduled events during the class. Selection of a segment from thetimeline causes the client on which the timeline is displayed (typicallythe instructor's client) to transmit one or more commands to cause thevideo conferencing system to implement the operations associated withthe segment 2120. In the illustrated example, segment 2120 ishighlighted to indicate that this section is currently being implementedby the online virtual conferencing system. Because this particularsegment 2120 is associated with conducting a poll with video of a singleparticipant being displayed in the central speaker region (as indicatedby the “1-up” indication), the selection of this segment (eithermanually by the instructor or automatically) causes the client device onwhich the segment is selected to transmit one or more commands to theonline video conferencing system to implement the poll using the “1-up”user interface arrangement. This may include, for example, generatingthe necessary data structures to collect the poll data and generating agraphical user interface which includes video of a single speaker in thespeaker region (e.g., the professor) and a region which includes one ormore poll questions to be answered by each participant.

In one embodiment, the dynamic state synchronization service 120described above in detail synchronizes the state of each client inresponse to receipt of the commands. For example, the dynamic statesynchronization service 120 may open records in the state database 115required to implement the online poll and may transmit synchronizationsignals to each of the clients participating in the online conference toensure that the virtual conferencing graphical user interface isconsistent across all of the clients.

In one embodiment, timing data may be associated with each of thesections and/or segments. For example, in FIG. 21A, the time allocatedto each section is displayed within the graphical elements, 2110 and2111, associated with each section (e.g., 10 minutes for each ofsections 2110 and 2111). In addition, in one embodiment, an elapsed timeindicator 2160 may be displayed showing the total amount of time whichhas elapsed during the class. The color of the elapsed time may beupdated to provide an indication as to whether the class is proceedingin a timely manner. For example, green may indicate that the class isproceeding on schedule or ahead of schedule, yellow may indicate thatthe class is proceeding slightly behind schedule (e.g., <5 minutesbehind), and red may indicate that the class is proceeding significantlybehind schedule. The system may determine how far the class hasprogressed based on the current segment highlighted within the timeline.

A notes section 2140 provides instructions to the professor related toeach segment. For example, the notes 2140 may provide generalinstructions related to the purpose and/or goals of the segment. Thenotes 2140 may also provide instructor notes from which the instructormay reference. A first graphical element at the bottom of the timelinemay be selected to display the notes and a second graphical button(e.g., located at the top of the notes) may be selected to hide thenotes.

FIG. 21B illustrates the exemplary graphical interactive timeline 2150integrated within the video conferencing GUI 2320, which may include thevarious virtual conferencing features described above such as a centralspeaker region, and video thumbnails of the various conferenceparticipants. In one embodiment, the instructor and/or an assistant tothe instructor are the only participants who are provided with access tothe interactive timeline 2150. The timeline may be hidden and broughtinto the display via a graphical user interaction by the instructor. Forexample, in one embodiment, the instructor may select a graphical buttonor other element to the right of the GUI or may select a designatedcontrol key on the keyboard to bring the interactive timeline 2150 intoview.

In one embodiment, when timeline events are active (e.g., in response tothe instructor selecting one of the segments 2120-2123), the event thatis highlighted automatically becomes disabled to prevent accidentalre-triggering of the event. When breakouts or polls are active in thetimeline, their main button will become disabled, but differentselectable actions may be provided. For example, for breakouts, a“message all” option may appear which opens the breakout sidebar withkeyboard focus on the message field (e.g., so the instructor may messageall students). In addition, an “end breakout” option may appear allowingthe instructor to end the breakout session. For polls, a “restart poll”option may appear to re-take a poll and an “end poll” option may appearto end the polling process. For a segment which provides an N-up (e.g.,placing N students in the speaker regions), N random students may beselected and/or N students may be selected based on specified selectioncriteria (e.g., picking the least talkative N students to feature). Inone embodiment, as soon as the event is no longer active the actionsdescribed above will disappear and the entire action will again becomeclickable.

FIGS. 22A and 22B illustrate how a lesson plan may be constructed andused to generate the graphical interactive timeline. In the embodimentshown in FIG. 22A, a lesson plan 2101 may be constructed in ahuman-readable format prior to the online class (e.g., by a team ofacademic advisors working for the online university, by the instructor,etc). FIG. 23 illustrates one such implementation in which a lesson plan2101 has been constructed using an online word processor (e.g., Google™Docs). One section 2301 is illustrated which indicates, in ahuman-readable format, a title, a time limit, and a start and stop timeassociated with the section. The first segment 2302 includes aspecification of the operations which need to be performed for thesegment including setting up a “2-up” view which includes video of theinstructor and a set of slides to be used for the segment. A script orset of instructions are also provided within the segment 2302. The topportion of a second segment 2303 is illustrated which indicates that sixbreakout groups are to be formed using 2-3 students in each groups basedon a particular attribute (e.g., frequency of participation in class,grade, polling results, or any of the other variables discussed above).

The lesson plan 2201 may be used to generate a machine-readablerepresentation of the lesson plan 2203. For example, in FIG. 21B,machine-readable lesson plan generation logic 2102 uses the lesson plan2101 to generate a machine-readable representation of the lesson plan2103. For example, the machine-readable lesson plan generation logic2202 may scan the lesson plan 2101 for certain keywords or fields andembed the data contained therein into the machine-readablerepresentation 2103. In an alternate embodiment, the machine-readablerepresentation of the lesson plan 2103 may be generated manually by auser (e.g., an academic team member) using the data contained in thelesson plan 2103.

Regardless of how the machine-readable representation of the lesson plan2103 is generated, in one embodiment, it is generated in a YAML format,a well-known human-readable and machine-readable data serializationformat (sometimes referred to as “Yet Another Markup Language” andsometimes using the recursive acronym “YAML Ain't Markup Language”).FIG. 24 illustrates an exemplary portion of a YAML representation 2401with arrows mapping section data and segment data to a graphicalinteractive timeline 2402. For example, “section 1” of the YAMLrepresentation 2401 includes a field for the section title and field forthe duration of the segment. Each segment includes data indicating atitle (e.g., “conduct poll”), an operation to be performed on the userinterface (e.g., “1-up”) and other pertinent information forimplementing the section on the video conferencing system (e.g.,specific panes and GUI features to be displayed). In addition, notes areprovided which may be used by the instructor during the class. Asmentioned, in one embodiment, the notes may be displayed beneath theinteractive timeline 2402.

Returning to FIG. 22A, timeline generation logic 2204 interprets themachine-readable representation of the lesson plan 2203 to generate thetimeline GUI and implement the underlying operations associatedtherewith. In one embodiment, the timeline generation logic 2204 isimplemented as program code executed on one or more virtual conferencingservers within the virtual conferencing service 100. The varioustimeline GUI features and associated functions are then streamed to theinstructor's client which may implement the timeline and other GUIfeatures within the context of a Web browser or conferencing applicationinstalled on the client. Alternatively, the timeline generation logic2204 may be executed directly on the instructor's client to generate thetimeline GUI and associated functions. In this embodiment, themachine-readable representation 2203 may be sent directly to theinstructor's client to be interpreted locally by the timeline generationlogic 2204. Of course, the underlying principles of the invention arenot limited to the particular location at which the program code forimplementing the timeline is executed.

In another embodiment, illustrated in FIG. 22B, a graphical designapplication 2204 is used to construct the timeline for each class andresponsively generate program code and/or a machine-readablerepresentation of the lesson plan 2212. One example of such a graphicaldesign application is illustrated in FIGS. 25A-C which includes a classtimeline region 2501 comprising a series of entries into which differentgraphical objects may be moved by the lesson designer to construct eachsection and segment. In the example shown in FIG. 25A, objects have beenmoved into a first set of entries 2510 (numbered 1-6) and a second setof entries 2511 (numbered 7-15) are available to receive new objects. Inone embodiment, the lesson designer may create a new section and/orsegment by clicking and dragging a new object into one of the openentries 2511. Different objects may be provided which representdifferent resources, tasks, and GUI configurations for each segment. Inthe example shown in FIG. 25A, the objects include new documents 2502,saved resources 2503 such as PDFs, notes, videos, breakout groups 2504,polls 2505, and screen sharing 2506. A virtual unlimited number of suchobjects may be created and made available for the lesson designer todesign each section and/or segment. In FIG. 25B, the lesson designer hasselected a second breakout object 2520 and is dragging the secondbreakout object towards the next open entry within the open entry region2511 (entry #7). FIG. 25C illustrates the second breakout object 2520positioned within the set 2510 of selected objects.

In one embodiment, each object provided within the graphical designapplication may have a set of parameters from which the lesson designermay select. For example, when selecting a new breakout group, adrop-down menu or other graphical selection structure may be provided toallow the lesson designer to select the parameters for the breakoutgroup (e.g., the number of participants per group, the resources to beused during the session, etc). Similarly, when conducting a poll, thelesson designer may be provided with a design widget to enter a questionand a set of possible responses. Various additional object-specificdesign features may be provided to allow the lesson designer to designeach section and/or segment.

In one embodiment, once the lesson designer has selected and configureda set of objects within the graphical design application 2211, thegraphical design application 2211 will generate program code and/or amachine readable representation of the lesson plan 2212 which may thenbe interpreted by timeline generation logic 2213 to generate thetimeline GUI and associated functions 2214 described herein. Asmentioned, the generation of the timeline GUI and associated functionsmay be performed on the virtual conferencing service or locally on theinstructor's client.

FIG. 26A illustrates one particular implementation where amachine-readable representation of the lesson plan 2103 is interpretedby timeline generation logic 2104 on a virtual conferencing service 100and the resulting timeline GUI and associated functions 2105 aretransmitted to the instructor's client 160. In this particular example,the timeline GUI and associated functions are implemented within thecontext of the conferencing GUI 162 executed within a browser orapplication 161 on the instructor's client 160. In one embodiment, thetimeline generation logic 2104 also establishes a database schemarequired to implement the functions of the timeline. The database schemamay be established, for example, to set up the resources and other staterequired to implement each section and segment within the timeline. Inone embodiment, the database schema is set up in accordance with thevarious operations and objects specified within the machine-readablerepresentation of the lesson plan 2103.

In addition, in the illustrated embodiment, validation logic 2601 isemployed which validates the machine-readable lesson plan 2103 prior togenerating the timeline 2105 and database schema 2600. For example, thevalidation logic 2601 may parse and analyze the machine-readablerepresentation of the lesson plan 2103 to ensure that no errors arepresent in the machine-readable representation. If the machine-readablerepresentation is in a YAML format, for example, the validation logic2601 may check to determine that the syntax used within the YAML file isvalid and may also check to determine that various resources such asfiles referenced within the YAML file exist in the locations specifiedfor the classroom.

As mentioned, once the timeline has been generated on the instructor'scomputer, the instruction may readily implement all of the operationsassociated with a segment by selecting that segment. A variety ofdifferent operations may be included within a segment including, by wayof example and not limitation, featuring a random or particularparticipant, or selecting a participant based on specified criteria(e.g., featuring the participant who has spoken the least in class thusfar, the participant who scored the highest on a recent quiz, theparticipant who answered a particular poll question in a particularmanner, etc). In addition, the segment may be configured to implement avariety of different graphical user interface elements such as featuringdifferent numbers of participants within the speaker region(s) (e.g.,1-up, 2-up, 3-up, 4-up, 5-up, 6-up, or 8-up for 1-8 featured speakers,respectively) or displaying one or more resources such as PDFs, Youtube®videos, links to web sites, word processing or presentation documents,spreadsheets, photos, to name a few. Other operations included in asegment may include, but are not limited to, conducting a poll,displaying poll results to the participants, comparing poll results ofmultiple polls or the same poll conducted more than once (e.g., oneconducted at the beginning of class and the other at the end of class),conducting a quiz, conducting a breakout session (e.g., selectingparticipants based on results of a poll), featuring breakout groups andtheir work-product (e.g., notes), pitting two students against oneanother in a debate, sharing the instructor's or a participant's screen,initiating an open discussion (with notes for the professor related tohow to direct it), and allocating a time period for independent work.

An exemplary database schema for the timeline 2600 is illustrated inFIG. 26B including a timeline component 2610, a TimelineSectioncomponent 2611 and a TimelineSegment component 2612 which define thedata structures used for the timeline, sections within the timeline, andsegments within each section, respectively.

After a user enters the timeline specification (e.g., as a YAML file orvia the user interface as described herein), it is sent to the server100, performs interpretation. Specifically, as mentioned above,validation logic 2601 may validate the YAML file or other machinereadable representation 2103, thereby ensuring that all references toresources, breakouts, polls, etc, exist and are accessible. In the caseof YAML file format, there are additional checks that the format isproperly adhered to. If there is a problem, it is reported back to theuser and the database is not updated.

In one embodiment, if validation passes, the timeline specification goesthrough a “normalization” step (e.g., implemented by timeline generationlogic 2104), whereby the human-interpretable YAML file is converted intoa form that is more uniform and thus simpler and more efficient for thecomputer to interpret. In one embodiment, the normalized form is writteninto the database 2600 using the schema shown in FIG. 26B.

In the illustrated schema, a “Timeline” 2610 consists of zero or more“TimelineSection”'s (connected through a foreign key). A“TimelineSection” consists of zero or more “TimelineSegment”s. EachTimelineSection has a “title” as well as “duration_in_seconds” forkeeping track of time during class and helping the instructor stay ontrack.

Each TimelineSegment has a generic “details” text field, which containsall of the information needed to display the user interface componentsof the timeline and perform the operations that are part of thattimeline segment. The “details” field is left generic because there is agreat deal of variety in the types of operations a TimelineSegment mightperform and, thus, a greater need for flexibility in what data isstored. Each TimelineSegment also has a “status” field, which indicateswhether this TimelineSegment (1) has not yet been used by the professor;(2) is currently in-progress in class; or (3) has already been used.This state is used to maintain the Timeline graphical user interface.

As mentioned above, different TimelineSegment operations have different“details” field formats (which, in one embodiment, are stored as jsoncode). An example “details” field for displaying on stage one studentwith email address student@minerva.kgi.edu and the last poll result forthe poll named “Reducing homelessness: Best practices” is shown below.

{  title: “Discuss poll result”,  op: layout,  number-of-panes: 2, duration: 120,  panes: [   {    type: “person”,    email:“student@minerva.kgi.edu”   },   {    type: “poll-result”,    name:“Reducing homelessness: Best practices”    instance-back: 0   }  ] }

In one embodiment, this normalized version would be created from auser-input YAML one example of which is:

Title: Discuss poll result

Duration: 2 m

Op: 2-up

Panes:

-   -   type: person    -   email: student@minerva.kgi.edu    -   type: poll-result    -   title: Reducing homelessness: Best practices

There are many user-input YAML files that would generate the samenormalized version. For example, a user could specify the full name ofthe person to feature as opposed to their email address. Converting tothe normalized version makes the client-side code that executes theTimeline easier and less error-prone to develop.

As mentioned, in one embodiment, the state synchronizationinfrastructure described above may be used (at least in part) toimplement the operations associated with each selected timeline segment.FIG. 27 illustrates one such example in which the instructor hasselected a segment, Segment 3, within a timeline GUI 2105. In response,the set of functions associated with Segment 3 are implemented insuccession and streamed to the dynamic state synchronization service 120on the virtual conferencing service 100. As in prior embodiments, thestate updates resulting from the execution of these functions are thenstreamed down to clients of each of the participants to the videoconference, including the client 160 of the instructor. For example, ifSegment 3 specifies that video of two speakers should be displayedalongside the results of a poll, then a sequence of commands to displaytwo speakers along with poll results will be streamed from the client160 and synchronized with all of the clients 140-160 via the dynamicstate synchronization service 120. In one embodiment, the commandstransmitted from the client 160 comprise the same set of commands thatwould result if the instructor manually set up the dual speakerconfiguration with poll results. The end result is that the local state145, 155, 165 on each client will be consistent and will display thedesired GUI and associated configuration.

In one embodiment, the instructor may log in to the virtual conferencingservice 100 using two devices and may use one device for implementingthe interactive timeline while using the other device for implementingthe standard conferencing GUI. FIGS. 28A and 28B illustrate theinteractive timeline implemented on a mobile device and tablet,respectively, and FIG. 28C illustrates the interactive timelineimplemented on a personal computer. In these embodiments, the user logsin to establish a first communication socket with the virtualconferencing service 100 using a first client device, and also logs into establish a second communication socket with the virtual conferencingservice 100 using a second client device. in one embodiment, the virtualconferencing service 100 will associated each of the two sockets withthe instructor's account and will process commands from either device ina similar manner. However, in one embodiment, the state updates from thedynamic state synchronization service 120 reflecting changes to thevirtual conferencing environment are transmitted to the client deviceimplementing the virtual conferencing GUI but not to the client deviceimplementing the interactive timeline.

System and Method for Decision Support in a Virtual Conference

As mentioned above, in some virtual conferencing systems, the meetingorganizer or moderator is provided with control over the current stateof the virtual conferencing system via a graphical control panel. Forexample, in a virtual classroom environment, the instructor is providedwith the ability to choose the current speaker or group of speakers.Once selected, video of the current speaker or group of speakers may bedisplayed within a central speaker region of the virtual conferencingGUI.

When an instructor (e.g., professor, teacher) chooses to call on astudent during in-class discussions, the choice of student to benefitfrom this learning opportunity may be unfair or inefficient. A choicemay be perceived to be unfair because humans are prone to variouscognitive biases (e.g., gender, or “availability” effects), which mayaffect a professor's decision in selecting a student to answer aquestion. An instructor's choice of a student may be inefficient becausethe educational value associated with calling on a student to answer aquestion varies and different students benefit to different degrees fromattempting to respond to the same question.

Web conferences outside of the context of a synchronous classroomenvironment suffer from a generalized form of this problem; conferenceleaders often intend to be fair and efficient in how they engage otherconference participants, but their attempts are impaired by varioushuman biases and inefficient choices. Furthermore, to the extent aninstructor or other moderator wants to be fair or efficient, he or sheneeds to expend both significant cognitive effort during theclass/meeting to track student behavior as well as time pre- andpost-class/meeting to prepare.

As a result of these limitations, some students (or participants) areafforded fewer learning and/or speaking opportunities than others andvaluable learning opportunities are missed.

One embodiment of the invention provides a web conference leader (e.g.,an instructor or other moderator) with information about whichparticipants (e.g., which particular students, meeting attendees) maybenefit most from active participation at a particular point in time.For example, a student may be called on to answer a particular questionduring a class, or a meeting attendee may be invited to speak at aparticular point during a conference. Using the techniques describedherein, speaking and learning opportunities are provided efficiently andfairly without bias.

While the discussion below will focus on a virtual classroom or“e-learning” environment to facilitate discussion of the presentapplication, one of ordinary skill in the art will appreciate that theunderlying principles of the invention may be extended to any virtualconference in which participants are invited to speak or otherwiseparticipate.

FIG. 29 is a block diagram illustrating components of an exemplarycomputer system which may be used to implement the various clients andservers described herein. The exemplary computer system may executeprogram code stored on a machine-readable medium to implement thevarious techniques described herein. Specifically, FIG. 29 shows adiagrammatic representation of a machine in the example form of acomputer system 2900 within which instructions 2924 (e.g., software) forcausing the machine to perform any one or more of the methodologiesdiscussed herein may be executed. In alternative embodiments, themachine operates as a standalone device or may be connected (e.g.,networked) to other machines. In a networked deployment, the machine mayoperate in the capacity of a server machine or a client machine in aserver-client network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. As is furtherdescribed below, this machine representation provides an exampleconfiguration to execute the processes described herein whether on theserver side and/or client side.

The machine may be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a set-top box (STB), a personal digitalassistant (PDA), a cellular telephone, a smartphone, a web appliance, anetwork router, switch or bridge, or any machine capable of executinginstructions 2924 (sequential or otherwise) that specify actions to betaken by that machine. Further, while only a single machine isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute instructions2924 to perform any one or more of the methodologies discussed herein.

The example computer system 2900 includes a processor 2902 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU), adigital signal processor (DSP), one or more application specificintegrated circuits (ASICs), one or more radio-frequency integratedcircuits (RFICs), or any combination of these), a main memory 2904, anda static memory 2906, which are configured to communicate with eachother via a bus 2908. The computer system 2900 may further includegraphics display unit 2910 (e.g., a plasma display panel (PDP), a liquidcrystal display (LCD), a projector, or a cathode ray tube (CRT)). Thecomputer system 2900 may also include alphanumeric input device 2912(e.g., a keyboard), a cursor control device 2914 (e.g., a mouse, atrackball, a joystick, a motion sensor, or other pointing instrument), astorage unit 2916, a signal generation device 2918 (e.g., a speaker),and a network interface device 2920, which also are configured tocommunicate via the bus 2908.

The storage unit 2916 includes a machine-readable medium 2922 on whichis stored instructions 2924 (e.g., software) embodying any one or moreof the methodologies or functions described herein. The instructions2924 (e.g., software) may also reside, completely or at least partially,within the main memory 2904 or within the processor 2902 (e.g., within aprocessor's cache memory) during execution thereof by the computersystem 2900, the main memory 2904 and the processor 2902 alsoconstituting machine-readable media. The instructions 2924 (e.g.,software) may be transmitted or received over a network 2926 via thenetwork interface device 2920.

While machine-readable medium 2922 is shown in an example embodiment tobe a single medium, the term “machine-readable medium” should be takento include a single medium or multiple media (e.g., a centralized ordistributed database, or associated caches and servers) able to storeinstructions (e.g., instructions 2924). The term “machine-readablemedium” shall also be taken to include any medium that is capable ofstoring instructions (e.g., instructions 2924) for execution by themachine and that cause the machine to perform any one or more of themethodologies disclosed herein. The term “machine-readable medium”includes, but not be limited to, data repositories in the form ofsolid-state memories, optical media, and magnetic media.

FIG. 30 illustrates an exemplary decision support module 3000 whichgenerates a prioritized list of students 3112 who would benefit the mostfrom participation, based on a set of criteria and student data 3110. Asdiscussed below, the criteria and student data may include one or morequestions (q) and learning outcomes (o) of the question with respect toeach student. The student data may also include variables such as thelength of time since the student has participated in the virtualclassroom.

In one embodiment, the decision support module 3000 is configured toquantify values of a question q, which is designed to promote a learningoutcome o, for each student. The decision support module 3000 comprisesa question outcome relationship identification module 3002, aparticipant evaluation and recommendation module 3004, and a userinterface module 3006. In one embodiment, the decision support module3000 is implemented in program code executed on the client of theinstructor. However, in another embodiment, the decision support module3000 is implemented on a server within the virtual conferencing service100 (i.e., and the decision results transmitted to the instructor'sclient).

In one embodiment, the question outcome relationship identificationmodule 3002 is configured to identify the relationship between aquestion and an underlying outcome, the participant evaluation andrecommendation module 3004 is configured to evaluate a participant'sperformance based on the identified relationship and recommend one ormore participants based on the evaluation, and the user interface module3006 is configured to generate a user interface to identify participantsfor speaking and/or learning opportunities (see, e.g., FIG. 31 andassociated text below). While illustrated within the context of thedecision support module 3000, the user interface module 2006 may beimplemented as a separate module which generates user interface featuresin response to the decisions made by the decision support module 3000.

Questions in some examples provided herein are teaching instruments. Forexample, instructors ask questions to facilitate teaching. Questions maybe asked to help the students achieve a learning outcome, to develop ahabit of mind, or to acquire a foundational concept. As illustrated, thedecision support module 3000 is configured to facilitate a webconference leader's decision making when selecting a student toparticipate in the virtual classroom. The present application refers tounderlying outcomes as o, questions as q, and students as s. In oneembodiment, the decision support module 3000 is configured to identifythe relationship between a question q and a student's output, which isthe student's performance with respect to the underlying outcome o. Amany-to-many relationship may exist between a question q and anunderlying outcome o, associated with the question q. It is noted thatthe decision support module 3000 may be embodied as a computer programproduct comprised of instructions, e.g., 2924, stored in a storage unitand/or memory, e.g., 2904, 2916, and/or 2922, an executable by aprocessor, e.g., 2902.

In various embodiments, a decision support module 3000 may generate ascore related to a student, by using a criterion function C, asillustrated in Equation (1):C:(o,s)→[0,1]

The question outcome relationship identification module 3002 may definea criterion function C to describe the relationship between a questionand an outcome. A criterion function C may be defined to include a setof factors. For example, a criterion function C may include a period oftime since the last time that a student s answered a question qinvolving outcome o correctly during class discussion. As anotherexample, a criterion function C may include a period of time since thelast time a student s answered a question q involving outcome ocorrectly in submitted homework assignments. Another criterion functionC may include a distance from mastery, which is a quantitativemeasurement indicating the difference of a student s from mastering anoutcome o (assuming that the students has not yet demonstrated masterybased on past performance on tests or quizzes). The distance frommastery is designed to provide a student with opportunities todemonstrate newly-acquired mastery or any incremental progress (e.g. ahigher score on a metric of mastery) based on the practice opportunityand professor feedback. In various embodiments, the distance frommastery is configured to be a metric including a set of factors such asa student's past attempts exhibited in synchronous settings (e.g. priorclass sessions), or a student's past attempts exhibited in asynchronoussettings (e.g. submitted assignments, discussion boards).

A criterion function C may further include a period of time since thelast time a student s has spoken in a class and/or may include thenumber of times that a student s fails in grasping the outcome o. Insome cases, an instructor may call on a student s who does not fullygrasp a concept and needs to have a basic understanding of the learningoutcome o.

The participant evaluation and recommendation module 3004 evaluates aparticipant's mastery of an outcome according to the defined criterionfunction C. A participant's performance in past sessions (which may berecorded) may provide a basis for evaluating the participant in apresent session. Various embodiments may record, store, and analyze,student participation in web conference session. For example, data maybe created to evaluate a student's performance according to a rubricdefining a set of evaluation rules provided by an instructor.

An instructor may take into account multiple criteria when determiningwhich student to select to participate. For example, in someembodiments, a composite criterion function C_(x) may be defined toinclude a set of individual criteria functions {C₁ . . . , C_(n)}. Acomposite criterion function C_(x) may be defined as the weighted sum ofa set of individual criterion functions and may be used to determine ascore for students according to an outcome o:C _(x)=Σ_(i=1) ^(n) w _(i) C _(i)  (2),where each of the criteria functions {C₁, . . . , C_(n)} has acorresponding weight {W₁, . . . , W_(n)}, andΣ_(i=1) ^(n) w _(i)=1.

Of course, in some implementations, the student selection process may besimpler than that described above. For example, in one embodiment, asingle criterion such as the length of time since each student lastparticipated or the frequency of participation of each student may beused as a sole variable for selecting a student. For example, theparticipant evaluation and recommendation module 3004 may generate theprioritized set of students 3112 based solely on one of these variables.

In the context of a single-criterion decision support module, a set ofstudents {s₁, . . . , s_(n)} may be ordered according to their C scoresbased on a learning outcome o associated with a question q, asillustrated in Equation (3). In various embodiments, the student withthe highest score according to a criterion C may be reported. Thestudent with the highest score is the one who, according to a decisioncriterion C, is expected to benefit the most from being asked a questionq focused on this learning outcome o.C _(sort):(o,{s ₁ , . . . ,s _(n)})→[s _(highest) , . . . ,s_(lowest)]  (3).

In the context of a multiple-criteria decision support module, a set ofstudents {s₁ . . . , s_(n)} may be ordered based on each individualscore according to a composite criteria function C_(x), according toEquation (4) below. One or more students with the highest score(s) arethe ones whom, according to the composite decision criteria, areexpected to benefit the most from answering the question that addressesthe learning outcome o.C _(x,sort):(o,{s ₁ , . . . ,s _(n)})→[s _(highest) , . . . ,s_(lowest)]  (4).

Various embodiments identify one or more students who are most likely tobenefit from answering a question addressing an outcome and may presenta metric to an instructor recommending one or more students to be calledon. The metric may include one or more students with the highestscore(s) based on an outcome associated with a question.

In one embodiment, information is presented to an instructor on-demandor in real-time within a virtual conference user interface asillustrated in FIG. 31. The user interface may present results of thedecision support module 3000 by highlighting a set of one or morestudents 3112-3117 having the highest scores for each decision criterionand/or highest composite scores. In particular, in the embodiment shownin FIG. 31, video thumbnails are displayed for all of the students and asubset of the video thumbnails 3112-3117 are highlighted usingmulti-colored circles. Once highlighted, the instructor may manuallyselect one of the highlighted user to participate (e.g., answer aquestion).

In one embodiment, the user interface module 3006 creates (or providefor display on a visual display or screen) the illustrated userinterface based on recommendations made by the participant evaluationand recommendation module 3004. In the illustrated example, the set ofvideo thumbnails 3112-3117 associated with recommended students arehighlighted to the professor 3110 based on the learning outcome 3120identified by the professor. A video image 3110 of the professor isillustrated in the center speaker region.

In various embodiments, a user (e.g., an instructor, a conferenceleader, a moderator) may specify an outcome (e.g., a learning outcome, aconference objective, a meeting objective) that is desired or needs tobe addressed. A set of recommendations is provided based on the outcome.An outcome may be defined in different ways. For example, a professormay indicate a learning outcome. A learning outcome may evaluate whetherhypotheses are based on plausible premises and assumptions, and arefalsifiable.

Before a class session begins, the professor 3110 may select a learningoutcome from a list provided. The selected learning outcome may beincluded in prepared class materials. The professor may pull it upduring a class from a dashboard of prepared class materials (e.g., suchas the interactive timeline 2402 described above), either on theirprimary computer display or a dedicated dashboard on a tablet or othermobile device (as shown in FIGS. 28A-C).

The professor 3110 may prepare specific questions before class andindicate the learning outcome associated with each question. Duringclass, decision support recommendations may then be generated by thedecision support module 3000 when a question is selected.

The professor 3110 may also specify a learning outcome during class,without advanced preparation. A set of learning outcomes may be providedin an order to make the selection straightforward. Minimum attention isrequired to make the decision while the professor 3110 leads a classdiscussion. Once again, the interface may be displayed on the primarydisplay or on a dedicated interface on a secondary display.

Various embodiments allow a user, e.g., the professor 3110, to selectfrom a set of criteria and/or a weight associated with a criterion. Inorder to calculate scores for students, e.g., 3112-3117, based on thespecified learning outcome, a set of criteria may be specified. If acomposite score is desired, per-criterion weights may be specified. Suchselections may be made before or during class. For example, theprofessor 3110 may select a criteria and/or a weight.

The professor 3110 may toggle on/off a criterion from a list of criteriaprovided before the class session begins (e.g., on the primary displayor a secondary display). The professor 3110 also may toggle on/off acriterion from a list of criteria provided during class, withoutadvanced setup. Moreover, for different questions, the professor mayactivate or deactivate a criterion. This interface may be displayed onthe primary display or on a dedicated view on a secondary display.

Various different types of information may be considered by the decisionsupport module 3000 to identify and prioritize a set of students whowould most benefit from participation. In many of the examples set forthabove, the criteria is based on specific questions and outcomes.Additional criteria may include the how frequently or how recently eachuser has participated in the class discussion. For example, oneembodiment of the decision support module 3000 may identity a set ofparticipants who have participated the least in class discussions and/ora set of participants who have not participated recently (compared withother participants). Using this information, the decision support module3000 may generate a prioritized list of participants with those userswho have participated the least towards the top of the prioritized listand those who have participated the most towards the bottom of theprioritized list. In other embodiments, the participation level of eachuser is just one criteria among several evaluated by the decisionsupport module 3000 when prioritizing the list of students. Like othercriteria, the participation level of each student may be assigned aweight which may be combined with the weights of other criteria toarrive at the final prioritization determination.

The number of recommendations for each active criterion may bespecified. For example, the instructor may specify the number ofstudents to be highlighted as potential candidates for participation.The number may be the same for all criteria or may vary by criterion. Ifa composite criteria scoring is desired, control over the weightsattached to each active criterion can be exposed. In one embodiment,weights may all be set to 1/n where n is the number of active criteria.Sliders (or other such controls) may be used to increase or decrease anyof the individual weights and the other weights can automatically adjustto compensate proportionally. For active criteria functions {C₁, . . . ,C_(n)} and current weights {w₁, . . . , w_(n)}, if w_(j) changed tow_(j′), the other weights may be adjusted in accordance with thedifference w_(j)−w_(j′) such that the sum of all weights is maintainedto be unity: Σ_(i) w_(i)=1.

Once an outcome is identified, recommendations may be presented to aninstructor or other moderator. In some embodiments, recommendations maybe determined, stored, and provided even before a class begins. By wayof example, based on specified criteria, recommendations may begenerated and integrated within the interactive timeline 2402 describedabove. In such a case, the instructor may only need to select therelevant segment from within the timeline 2402 to view the set ofrecommended students 3112-3117.

While one specific implementation is illustrated in FIG. 31, in whichdifferently-sized highlight elements are displayed over certain videothumbnails 3112-3117, decision support recommendations may be presentedto a user in a wide variety of ways. For example, recommendations can beconveyed to an instructor by visually modifying or styling video orphotographic representations of students through: (1) change inbrightness, opacity, saturation, and/or contrast of the video orphotograph of relevant students; (2) change in size, color, or opacityof a border or shadow around the video or photograph of relevantstudents; (3) addition of a pulse or wiggle animation of a border orshadow around the video or photograph of relevant students; (4) increasein size/dimensions of the video or photograph of relevant students; (5)addition of an icon or badge on or near the video or photograph ofrelevant students; (6) a word or phrase appearing below or otherwiseassociated with the video or photograph of relevant students; or (7) achange in position of a photograph (e.g., slightly elevated on y-axis),or change in ordering of a row of photos (e.g., arranging students fromleft to right in a descending order).

Students that are evaluated to benefit from a learning opportunity todifferent degrees may be visually modified differently. For example, inFIG. 31, a more pronounced (i.e., larger) highlight is used for thestudent identified in video thumbnail 3112 compared to the studentidentified in video thumbnail 3113 (indicating that the student in 3112may benefit more from participation based on the results generated bythe decision support module 3000).

In one embodiment, students who demonstrate a lower mastery of a subjectmay be presented more visually distinguishable than students whodemonstrate greater mastery. A student may be presented most visuallydistinguishable when the student is determined to demonstrate the leastamount of mastery of the underlying outcome associated with a question.Students 3112, 3115-3116, and 3117, in the illustrated example, are morevisually distinguishable than students 3113-3114.

For instance, the highlighted area around students 3112, 3115-3116, and3117 are larger and the color is brighter than the highlighted areaaround students 3113-3114. Via this visual presentation, the professor3110 may be notified that among the students being recommended3112-3117, students 3112, 3115-3116, and 3117 have demonstratedrelatively less mastery. As such, the professor 3110, with very littlecognitive effort, may make a decision in picking a student to answer thequestion. In one embodiment, the professor may pick a student byselecting the video thumbnail or other icon associated with thatstudent.

The relevant criteria may be identified and presented to a user.Recommendations may be made based on at least one criterion, thecriteria used as a basis for each recommendation may be identified andpresented to a user. Criteria that are used may be presented to a userby visual modification. A visual style modification can display the nameof and/or short identifier for one or more criteria by which the studentwas recommended, either always or upon a trigger action like a mouseclick, hover, or tap.

When there are multiple displays, a live-updating display ofrecommendations may be visible for all active criteria for all learningoutcomes of interest. All recommendations are always visible for alloutcomes of interest. A user does not need to select a learning outcometo view recommendations. For example, a grid of student names or photosmay be displayed. Each row is associated with one learning outcome andeach column is associated with one decision criterion. In each cell, avisual representation of the top students being recommended for thecombination of outcome and criterion is displayed.

When multiple criteria are selected but only one outcome is specified, alist of top students per criteria are distributed across multiple rows,such that only one student is displayed in a cell. When several outcomesare selected but only one criterion is specified, a list of students ineach cell may be distributed across multiple columns, such that only onestudent is displayed in a cell.

A method in accordance with one embodiment of the invention isillustrated in FIG. 32. The method may be implemented within the contextof the system architectures described above, but is not limited to anyparticular system architecture.

At 3201, criteria and student data are evaluated to generate a set ofprioritized student participants (i.e., ordered based on relativepriority). As mentioned, in one embodiment, multiple weighted criteriamay be used to perform the prioritized ordering. Variables such as thespecific question, the learning outcome for the question, and the lengthof time since each student has last participated may be evaluated togenerate the ordering.

At 3202, representations of the prioritized students may be highlightedwithin the instructor's graphical user interface based on the relativeprioritization of the students. For example, in the embodiment shown inFIG. 31, the highlight graphics surrounding some of the prioritizedstudents 3112-3117 are bolder and/or larger to reflect theprioritization. Of course, various other techniques may be used toconvey the prioritization including simply ordering the representationsof the students based on priority (e.g., higher priority to the left andlower priority to the right), displaying a numerical value, and/or usingcolor-coding.

At 3203, an indication of one or more prioritized students selected bythe instructor/moderator are received (e.g., by program code executed onthe instructor's client and/or the virtual conferencing service 100). Asmentioned, the instructor may select a student simply by clicking on thehighlighted representation of that student within the graphical userinterface.

At 3204, signals are generated to rearrange the user interfaces of allof the virtual conference participants in response to the instructor'sselection. For example, if the instructor has selected a particularstudent to address a question, then that student may be moved into thecenter speaker region of the graphical user interface. In oneembodiment, the rearrangement of the user interfaces is accomplishedthrough the dynamic state synchronization service as described above.For example, the local state of the instructor's user interface mayfirst be modified to reflect the selection. The state changes may betransmitted to the dynamic state synchronization service from theinstructor's client from which they may be propagated to all clientsparticipating in the virtual conference (e.g., using thepublish/subscribe techniques described above).

System and Method for Tracking Events and Providing Feedback in aVirtual Conference

As mentioned above, students in a virtual classroom can benefit fromformative feedback on their contributions during classroom discussions.Such feedback requires identifying, classifying, and/or assessing eachcontribution (e.g., spoken contribution, written contribution) by eachindividual participant, which may be time consuming and impractical.Notwithstanding the pedagogical value of formative feedback andassessment, the time and resources required in providing such feedbackand assessment to participants can prevent or diminish this learningopportunity for participants.

One embodiment of the invention actively captures “events” related tothe activity of each student during the course of a virtual conference.These events may then be stored along with timing data to indicate thepoints in time during the course of the virtual conference when theevents occurred and information associating each event with particularparticipant. An instructor or other moderator may then review theactivities of a given participant by filtering the events associatedwith that participant from the events associated with all otherparticipants. In one embodiment, the timing data may be used to reviewthe events within the context of the overall virtual conference (i.e.,at the specific times at which the events occurred). In this manner, theinstructor or other moderator can efficiently review the contributionsof each student and provide formative feedback. In one embodiment,identified contributions may be classified and assessed according to thegoals defined by the instructor.

While the embodiments described below focus on a virtual classroom or“e-learning” environment to facilitate discussion of the presentapplication, one of ordinary skill in the art will appreciate that thepresent application can be used to provide feedback and support decisionmaking in various other forms of virtual conferences.

FIG. 33 illustrates an example feedback provision module 3300 whichevaluates the video conference contributions 3310 of each participantand responsively generates sets of assessed and categorizedcontributions 3312. In one embodiment, the assessed and categorizedcontributions 3312 includes data associating each contribution with atleast one participant and timing data indicating a point in time atwhich the contribution occurred during the virtual conference. Thefeedback provision module 3300 may be embodied as program code (e.g.,software instructions) that may be read into the memory 2904 andexecuted by the processor 2902 of the computer system 2900 asillustrated in FIG. 29. The exemplary feedback provision module 3300comprises a contribution identification module 3302, a contributionclassification module 3304, and a contribution assessment module 3306.Briefly, in one embodiment, the contribution identification module 3302identifies a participant's (e.g., a student's) contribution during thevirtual conference, the contribution classification module 3304classifies the participant's contribution, and the contributionassessment module 3306 assesses a participant's contribution andgenerates assessment data.

Turning first to the contribution identification module 3302, aparticipant's action may be identified as a contribution according to aweb conference leader's (e.g., the instructor's) definition or accordingto some predefined criteria (e.g., implemented in a rubric). Varioustypes of actions may be identified and recorded for assessment purposesincluding, by way of example and not limitation, a participant speakingduring a class, a participant typing a comment during a class, a commentprovided by a participant and publicly spotlighted by an instructorduring a class, a participant's response to a poll or a quiz questionduring a class, a participant formally asking a question during a class,or any of the above actions in a group discussion. As mentioned abovewith respect to FIG. 20, high quality audio and/or video clips 2000 ofeach student's participation may be recorded locally on each client 130and subsequently archived within a multimedia storage service 190. Inaddition, a participant's action may be recognized from the recordedaudio and/or video clips 2000, as further described herein.

An event may be recorded when the contribution identification module3302 identifies a participant's action as a contribution of interest toa web conference leader or assessor. In one embodiment, upon detectingan action of interest, a record is created to store the action. Forinstance, when a user U takes an action A at time T and the action A isrecognized as a contribution, other participants may be notified thatthe action A taken by the user U and an entry may be created to recordthat user U took an action A at time T. The participants may be notifiedthrough the dynamic state synchronization service 120 (e.g., using apublish/subscribe mechanism described above) and a record of the eventmay be created and stored within an event log 3331 on a log server 3330(e.g., on the virtual conferencing service 100 and/or on each of theindividual clients).

In some embodiments, all actions by all participants are recorded andstored by the contribution identification module 3302 according to areference time point. That is, all events are recorded and loggedincluding those that may not be of interest to a web conference leader.As such, an event type that was not initially perceived to be ofinterest when an event occurred may be retroactively incorporated if itis later determined to be of interest. For example, when a user U takesan action A at time T, a server may be notified by a message to publishthis action A at time T. The other participants may be notified and alog entry may be created to record that the user U took the action A atthe time T. The log entry is thus associated with a timestamp.

As mentioned, an event of interest may be recognized by the contributionidentification module 3302 when a participant's action is detected anddetermined to meet a web conference leader's definition. For example, inthe case of speaking contributions, a “push-to-talk” model or a“record-when-speaking” model as previously described may be used todetect a participant's contribution which is of interest to theconference leader. In the “push-to-talk” model, a participant activatesa microphone and features his or her video by a trigger action (e.g.,holds a specific key on a keyboard, toggle a button) Such a“push-to-talk” model may generate a time-stamped“spoken-contribution-started” event. Audio may be recorded from thestudent's microphone when the triggering action is detected (e.g., a keyis pressed). Recording of audio may be stopped or terminated whenanother triggering action is detected (e.g., a key is released) and a“spoken-contribution-ended” event is generated. The recorded audio maybe uploaded and archived on a server (e.g., on the multimedia storageservice 190) along with the starting event and the concluding event. Assuch, records of when a participant starts and finishes speaking and anaudio clip of his or her contribution may be created. The duration ofthe participant's spoken contribution may also be determined andrecorded. In the “record-when-speaking” model, a participant's audioinput is monitored for sound that is likely to be recognized as humanspeech (e.g., volumes above a threshold).

In one embodiment, a speech recognition module or application translateseach participant's verbal contribution into text—either during theconference or as a post-processing operation following the conference.The speak recognition operations may be performed on recorded speech atthe multimedia storage service 190 (e.g., after all captured audio clipshave been uploaded) or on each individual client 130 (e.g., directlyfrom the captured audio clips 2000). In this embodiment, the text may bestored and associated with the speaking event which was used to generatethe text (e.g., using the same set of timestamps).

Records may be created to log each event within the event log 3331, forexample, “spoken-contribution-started” or “spoken-contribution-ended”events may be recognized and recorded each time the threshold (e.g., −50dB) is crossed and/or each time the user holds down the “push to talk”button. Every identified event is associated with reference time pointsin the audio/video recording, of which one reference time pointindicates the starting point of the identified event and the otherreference time point indicates the completion point of the identifiedevent. In one embodiment, a reference event is created immediatelybefore the recording such that the time point to which the referenceevent corresponds is used as an offset to determine the reference timepoints for an identified event. For example, as illustrated in FIG. 34,when a class starts, event logging begins at time A (e.g., record starttime). The instructor requests audio/video recording at time B, and therecording is enabled at time C. When an event is logged at time D, thelog of the event is synchronized within the event log 3331 with thevideo recording of the event by associating it with timestamp “TS=(D−C)”in the recorded audio/video.

In one embodiment, the contribution classification module 204 classifiescontributions made by participants according to one or more predefinedevaluation rubrics or evaluation rubrics defined by a conference leader.A conference leader may be able to specify an outcome and a rubric forevaluating each contribution making contribution classificationefficient and accurate. One embodiment of the contributionclassification module 204 allows a user to non-linearly navigate atethered video timeline and event stream. In addition, the user mayflexibly filter events to separate events that are contributions fromthose that are not contributions in the event log 3331. A user mayselect one or more learning outcomes of events from a larger set ofoutcomes. An outcome may be an objective to be achieved in a webconference session. A participant's contributions may be classifiedbased on the types of the contributions (e.g., speaking, typing,writing, drawing, outlining, etc.). Each contribution may be associatedwith a set of contribution attributes such as the type, the timestamp,the duration, and/or the content, etc.

A non-linear navigation that employs at least two mechanisms to navigatea video recording may be provided. FIG. 35A illustrates an exemplaryuser interface which includes a media player component 3501 fordisplaying video and rendering audio of a recorded video conference andan event list 3502 for selected portions of the recorded virtualconference. As illustrated in FIG. 35A, a video may be played, paused,and searched using a video controller (or associated keyboard shortcuts)in the media player 3501. In addition, one or more events may beselected from an event list 3502, via a user interface controller (e.g.,by clicking or tapping on an event in the list 3502 or triggering via akeyboard shortcut). In one embodiment, each item in the event list 3502is an event associated with an event type, event description, eventtimestamp, event creator/actor, and/or event type-specific metadata anddetails. In one embodiment, when a video is navigated using one of thenavigation mechanisms (e.g., moving the navigation bar within the userinterface of the media player), a corresponding event within the eventlist may be automatically highlighted. For example, when playback isdragged to a new point in the video, the event with timestamp beforethat time point or the closest to that time point is selected. Whileplaying a video, when the playback passes over the timestamp of anevent, that event is visually highlighted in the event list 3502 (e.g.,changed to be distinguishable from the other events). When an event isselected from the event list 3502 via a triggering action, the videoplayback jumps to the time corresponding to the timestamp of theselected event. While the media player 3501 is displayed above the eventlist 3502 in the illustrated example, the media player 3501 and theevent list 3502 may be positioned in other configurations (e.g., side byside).

In one embodiment, each event within the event list 3502 may bedisplayed in accordance with the type of event. In the example shown inFIG. 35A, different events are identified using different icons (e.g.,speaking events include a speaking bubble icon, typing events include akeyboard icon, etc). In addition, color coding may be used to indicatewhether an event in the list is a speaking event, typing event,bookmarked moment, etc. The color coding may be set to the left, rightor across the entire event within the event list 3502. Of course,different events may be identified in various different ways while stillcomplying with the underlying principles of the invention.

In addition, events within the event list may be filtered byparticipant, event type and/or event content. Examples of filtering GUIfunctions are illustrated in FIGS. 35B and 36 and described in greaterdetail below.

FIG. 35B illustrates another embodiment of an event list 3503 whichvisibly includes event details with formatting that varies by eventtype. For example, the bars shown in FIG. 35B each have a differentcolor corresponding to a different event type (including bookmarkedevents, discussion events, slides, and questions). A user may configurethe display of certain event types via the user interface control 3504which includes selection elements (e.g., selectable check boxes)allowing the user to apply a set of filters in accordance with theselected elements. For example, if the user were only interested indiscussion events (e.g., speaking events), then the user would place acheck only in the discussion selection element.

FIG. 36 illustrates another example event filter control 3604 throughwhich a user may control and select a subset of events associated with avirtual conference. The illustrated event filter control 3604 includes afirst section 3601 in which the user may specify the types of events tobe displayed which includes spoken comments, typed comments, featuredcomments, bookmarked moments, and liked moments. Some of the event typesare configurable for further fileting. For example, the user may selectspoken comments above a user-configurable duration (e.g., 7 seconds inthe example) and may select typed comments above a certain length (e.g.,10 words in the example). The second section in the event filter control3602 allows the user to filter based on the identity of one or morestudents. Here, using radio buttons, the user may select the entireclass, an individual student, or the user herself. A third section 3603allows the user to choose between assessed contributions (e.g., thosefor which feedback has been provided), unassessed contributions, orboth.

As mentioned, when users take actions that are recognized ascontributions, events are automatically generated and stored within theevent log 3331. In one embodiment, metadata related to those actions isalso associated and stored with the events. For example, an event may beassociated and stored with metadata defining an event type (e.g. spokencomment, typed comment, bookmarked moment), a type-specific eventduration (e.g. number of words in written comment, number of seconds inspoken comment), an event actor (i.e. which person took the actionresulting in the event entry), and/or the number of assessments of theevent. In this embodiment, the event filter control 3602 provides theuser with options to use the metadata as a basis for filtering orsorting events. Different categories of filters may be provided toselect events to be displayed. Upon selection by the user, the eventlist displays events that match the conditions defined by the filters.If an event does not match conditions specified by one or more of thefilters, the event is not displayed.

Event filters may include an event type filter, a type-specific filter,an event actor filter, or an event assessment filter. Examples of theevent type filter and type-specific filter are illustrated in section3601 of the event filter 3604; examples of an event actor filter areillustrated in section 3602, and examples of an event assessment filterare shown in section 3603. By way of example, an event type filter mayinclude spoken contributions, typed contributions, instructor-featuredcontributions, or bookmarked moments. A type-specific filter may includethe minimum duration (e.g., time duration, word counts) of aparticipant's contribution (e.g., spoken contributions, typedcontributions). An event actor filter allows a user to select eventsmade by everyone, by a specified/named participant, or by theviewing/current participant. An event assessment filter allows a user toselect events that have been assessed or have not yet been assessed. Anevent list 3502, 3504 may be updated in real time when a filter isenabled, disabled, or modified. An indicator of the number of eventsincluded in a list may also be displayed.

FIG. 37A illustrates an example user interface 3701 for selecting alearning outcome. A web conference leader may select a learning outcometo associate with an event thereby choosing the most efficient path fortheir particular situation. Leaning outcomes may be organizedhierarchically. A user may select a learning outcome by clicking a mouseor trackpad, tapping on a touch-screen input, or typing a keyboardshortcut or by making a series of actions. For example, an instructormay select a learning outcome from a domain of 125 learning outcomes bymaking a series of three 5-option selections.

FIG. 37B illustrates a user interface 3702 including a learning outcomeat the bottom of an organizational hierarchy which may be “pinned” inadvance of the web conference for quick access. For class discussionsthat are focused primarily on a small number of learning outcomes,providing fast shortcuts to these outcomes can avoid repeating the samemulti-step selection process many times for a single assessment session.

FIG. 37C illustrates an example user interface 3703 in which a searchfield 3704 with an auto-complete feature helps a web conference leaderfind learning outcomes based on a keyword, which can efficientlyfacilitate selection via the hierarchical structure. When an instructortypes in the text field, only items that match with the words typed aredisplayed. As such, the instructor may locate the appropriate outcomemore easily.

FIG. 38 illustrates an evaluation rubric 3800 from which a student'sevaluated contributions may be selected. In one embodiment, thecontribution assessment module provides these options to the instructorand compiles the instructor's selections. After the identified eventsare classified according to learning outcomes, each student may beassessed with respect to the learning outcomes. The instructor mayiterate over the list of students and view filtered sets ofcontributions made by each student labeled with each relevant learningoutcome. The instructor may then evaluate the student's contributionsand provide a feedback based on the student's contribution and theobjective (e.g., a learning outcome).

FIGS. 39-41 illustrate example user interfaces for evaluating andproviding feedback on a participant's contribution. This embodimentincludes the event filter 3604 for filtering events within an event list3502 as described above, and also includes a feedback region 3902 forperforming an evaluation and providing feedback. As illustrated, theright pane of the user interface may be used to provide feedback via thevarious user interface components 3701-3704, 3800 described above forselecting learning outcomes and evaluation content. For example, in FIG.39, an instructor may review a participant's contributions by selectinga contribution from the filtered event list 3502 and providing feedbackon that contribution from user interface component 3702 within feedbackregion 3902. As illustrated in FIG. 40, the instructor may also reviewand provide input via an evaluation rubric 3800 which provides aguidelines for evaluating a contribution (e.g., providing a ranking from0-4).

Subsequently as illustrated in FIG. 41, the instructor may providespecific comments to a participant related to an event via a commententry window 4100. The evaluation result may then be further appendedwith the additional comments from the instructor. The evaluation resultmay be submitted to a classroom grade book and the student may beimmediately notified of the feedback upon logging in to the virtualconferencing service 100. In various embodiments, in a period of timespecified by an instructor, a summary of a participant's contributionsmay be provided. For example, data analytics such as the frequency ofthe participant's contributions, evaluations of each of theparticipant's contributions, or a comparison of a participant to otherparticipants in a session may be provided. Feedback may then be providedbased on the participant's contributions and data analytics.

FIGS. 42-43 illustrate example user interfaces for a participant toreview evaluations submitted by an instructor. As illustrated, aparticipant may review evaluations and/or additional comments byselecting a contribution event from the event list 3502. Evaluations andadditional feedback from an instructor may be displayed to theparticipant.

Some embodiments of the invention provide peer assessment among webconference participants to facilitate peer evaluations. The same orsimilar techniques described herein may be employed for peer assessment.For example, peers may be provided with an event list 3502, media player3501, and feedback region 3902 using the the same GUI features asdescribed above. Thus, a web conference participant may be evaluated byboth the web conference leader and other web conference participants.The goals may also be defined by both the web conference leader and webconference participants. A web conference participant's contributionsmay be identified and classified according to an outcome defined by theweb conference leader and the other web conference participants. A webconference participant's contributions may be evaluated according to oneor more evaluation rubrics and the objective. A participant may beevaluated in every session, in multiple sessions collectively, or over aperiod of time.

When no evaluation rubric is defined, contributions or events that areidentified may be provided as appropriate attachment points to anchoropen-ended discussions about specific contributions. The use cases varybased on permissions and visibility of comments on contributions. Forexample, if only a web conference leader can attach comments, only theweb conference leader is provided with appropriate context to makecomments. In some cases, both a web conference leader and web conferenceparticipants can comment and respond, but each web conferenceparticipant is only permitted to see comments made on his or her owncontributions. A venue for asynchronous private discussions andfree-form formative feedback is thus provided. In some other cases, botha web conference leader and web conference participants can comment andrespond and all comments visible to everyone, and a public venue forcontinuing discussions after a web conference session ends is provided.If only web conference participants can comment on any entry but onlythe individual whose contribution is being commented on can see thecomments, a structured note-taking space for the participant to annotateand reflect on the comments is provided. The comments may be madeavailable for different purposes. For example, web conferenceparticipants and a web conference leader may be able to mark eachcomment as public, prof-student only, or entirely private.

Further learning opportunities are provided in addition to the goal ofproviding students with formative feedback on discussion participation.An event list may be filtered to include all events or contributions ofa student during any class session in a course, with summary statisticsquantifying the student's participation over the duration of the course.Summary statistics may include graphs and trend evaluation scores overtime. An event list may be filtered to include events showing allcontributions made by the student that are associated with one objectivein any course during a semester, providing the student with acomprehensive view of their engagement with the objective. A student mayreview events showing all contributions associated with a particularlearning outcome, made by the student who received the top score for thelearning outcome in that class session. As such, the student may findexamples of outstanding contributions for the particular learningoutcome.

The disclosed configuration beneficially includes eliminating orreducing cognitive biases and the amount of cognitive effort involved inmanaging and providing feedback on web conferences. Various embodimentsdescribed herein may provide a web conference leader with evaluationsand recommendations of web conference participants to facilitate the webconference leader's decision in engaging web conference participants ina session and providing feedback. Evaluations and recommendations of webconference participants may be based on an objective identified by a webconference leader. A criterion function may be defined to describe therelationship between the objective and a web conference instrument(e.g., a question). The criterion function includes one or more factorswhich may be configurable by the web conference leader. Web conferenceparticipants may be evaluated by using the criterion function andrecommended based on the evaluations. Using the techniques describedherein, a web conference leader may make a web conference session fairerand more efficient with little cognitive effort and preparation during,before, and after a web conference session. Web conference participantsthereby may be afforded with equal amount of learning or speakingopportunities.

System and Method for Discussion Initiation and Management in a VirtualConference

Discussion between web conference participants can facilitateinteractions among web conference participants and be used as a methodfor teaching or other purposes. However, during a discussion session, itis difficult to engage all the web conference participants. For example,participants who are not active participating (i.e., observers) maybecome disengaged and not actively contribute. Furthermore, recordingparticipants' feedback may be challenging.

Referring now to FIG. 44, it illustrates a flow diagram of an exampleprocess of creating and maintaining a debate in web conferences. Theflow diagram corresponds to functional instructions, e.g., 2924, thatmay be stored on a computer readable storage medium, e.g., 2904 and/or2922, and executable by a processor, e.g., 2902. A web conferencemoderator (e.g., an instructor in a virtual classroom) 4401 and all theparticipants of the web conference (e.g., debaters I-N 4402-4403 and theobservers 1-N 4404-4405) are engaged during the entire discussion period(e.g., logged in to a virtual conference). For example, the webconference moderator 4401 may initiate a discussion through their clientand each participant may be engaged in the discussion through their ownclient computer system (e.g., 2900).

The web conference moderator 4401 may create a debate at 4410 bychoosing two or more participants of a web conference to debate eachother. In addition, at 4410, the web conference moderator 4401 maychoose a debate topic and one or more rubrics from a database of storedrubrics which define a framework according to which a debater can beevaluated (e.g., defining the basis for evaluation of the debaters). Thedebate topic and the rubrics selected by the moderator 4401 may be madeavailable to the debaters 4402-4403 and/or observers 4404-4405. Theobservers 4404-4405 (i.e., the participants who are not chosen to bedebaters) may actively evaluate each debate and their evaluations may bemade visible in real-time. Further analysis (e.g., disparity,variability) of the evaluations may be provided in real-time. Inaddition, during a discussion session, the moderator 4401 or any webconference participant may create a bookmark to mark a point in a debateor other discussion which may then be revisited by selecting thebookmark (e.g., via the event list described above).

As the debate is initiated at 4411, the debaters and observers areprovided with the ability to view the debate topic and relatedparameters (e.g., the rubric for evaluating the debate) and, at 4412,the observers may actively score the debate in accordance with thedebate topic and rubric. in one embodiment, the scores submitted byobservers are collected from the observers in real time and may bevisualized during the debate and/or stored for later analysis.

FIG. 45 illustrates an example discussion support module 4500 to supportdiscussions in web conferences by generating a discussion configuration4512 comprising the data structures and GUI features required toimplement the discussion in response to a set of discussion parameters4510 specified by the virtual conferencing moderator (e.g., number ofdebaters, rubrics for evaluation, debate topics, etc.). The discussionsupport module 4500 can be embodied as program code (or softwareinstructions, e.g., 2924) that can be read into the memory 2904 andexecuted by the processor 2902 of the computer system 2900 illustratedin FIG. 29. The computer system on which the discussion support module4500 is implemented may be a server on the virtual conferencing service100 or one of more client computer systems of participants ormoderators. In one embodiment, the discussion support module 4500comprises program code reflected in the current state 135, 145, 155, 165managed on each client 130, 140, 150, 160 using the statesynchronization service 120 discussed above (e.g., which, in oneembodiment, utilizes a publish/subscribe mechanism to maintain aconsistent state on each client). However, the underlying principles ofthe invention are not limited to any particular location or mechanismwithin the virtual conferencing system for implementing the discussionsupport module 4500.

The illustrated discussion support module 4500 comprises a discussioncreation module 4502 and a discussion maintenance module 4504. Thediscussion creation module 4502 creates a discussion involving a groupof participants in a web conference session using the specified set ofdiscussion parameters 4510. The discussion maintenance module 4504maintains and updates the necessary data structures and GUI featuresamong all participants including those do not actively participate inthe discussion in a web conference session. In addition, in oneembodiment, the discussion maintenance module 4504 compiles feedbackfrom the virtual conferencing participants and/or the moderator duringand/or after the discussion (e.g., such as voting results on a debate asdescribed below).

In one embodiment, the discussion creation module 4502 allows a webconference leader to create a discussion in a web conference session byspecifying a set of discussion parameters 4510. Various differentdiscussion parameters may be specified by the moderator to create adiscussion including, by way of example, and not limitation, thediscussion topic, the number of participants involved in the discussion,the discussion participants, the position that each side of thediscussion will take, and evaluation criterion according to which adiscussion participant may be evaluated. A discussion may involve a setof participants against another set of participants, where a set ofparticipants comprises one or more participants. The evaluation criteriamay include an evaluation rubric which may be made available to thediscussion participants including the debaters 4402-4403 and theobservers 4404-4405. The evaluation rubric may be determined by the webconference moderator 4401 based on an objective. For example, in theclass format of a web-meeting, an evaluation rubric may be aligned witha learning outcome for the class. In a business context, an evaluationrubric may be aligned to the criteria used to make a decision (e.g., aweighted decision matrix).

In one embodiment, the discussion support module 4500 may be used incombination with the timeline GUI and associated functions 2105described above to initiate a debate. For example, a segment in thetimeline GUI may specify when the debate should occur and the variousparameters 4510 to be associated with the debate. in such a case, theinstructor (or other moderator) may initiate the debate by simplyselecting the graphical debate segment within the timeline 2105. Inaddition, the decision support module 3000 described above may be usedto identify the set of debaters 4402-4403 to participate in the debate.For example, upon initiation of the debate (e.g., from the timeline orotherwise), the decision support module 3000 may identify those studentswho would most benefit from participation in the debate based on thecriteria and student data 3110 as described above (e.g., such as howrecently each student has contributed to the topic or in a priordebate), and provide the list of students as parameters 4510 to thediscussion support module 4500. Using these techniques, the moderatormay initiate the debate fairly and efficiently with little cognitiveeffort.

FIG. 46 illustrates an example user interface which may be utilized byan instructor or other moderator to initiate a debate. The illustrateduser interface includes a pop-up menu 4600 allowing the instructor toselect from among a set of saved debates or to initiate a new debate.The user interface presents, in this example view, all the webconference participants as thumbnail images 4601 across a top of theuser interface. A web conference leader may select two or moreparticipants to participate in the debate by selecting the thumbnailimages 4601 corresponding to the participants. The example userinterface also illustrates in this view a page 4602 from a book that mayform the topic that the selected participants will discuss, for example,in the form of a debate. The pop-up menu 3800 on the right in theexample view controls the parameters of the debate and allows for thecapture of data from the debate that corresponds to a rubric to be used.

FIG. 47 illustrates additional features of the example user interfacefor creating a discussion. In one embodiment, the new pop-up window 4700may be automatically generated in response to the instructor or othermoderator selecting a new debate from pop-up window 4600. In particular,the illustrated pop-up window 4700 includes data fields in which theinstructor or other moderator may enter a new debate topic and a set ofdebate arguments (e.g., one or more positions that a discussionparticipant may take). The discussion topic and the discussion positionmay be related to an objective specified by the web conference leader.

FIG. 48 illustrates a new pop-up window 4800 which may be generated inresponse to the instructor entering data and selecting the “Next” buttonfrom the pop-up window 4700 in FIG. 47. A web conference leader mayspecify criteria for evaluating a discussion participant which isdefined according to the web conference leader's objective. As such,evaluation of discussion participants is fair and uniform. In theexample shown in FIG. 48, the selection options include (I) thinkingcritically, (II) thinking creatively, and (III) communicatingeffectively. An evaluation criterion may include an evaluation rubric.An example evaluation rubric is illustrated in Table 1 below.

TABLE 1 0 - 1 - 2 - Barely 3 - Adequate: 4 - Good: 5 - Excellent: NoUnsatisfactory: Adequate: Student's work Student's Student's work DataStudent's work Student's work shows basic work shows shows shows noshowslimited comprehension consistent use comprehension comprehensioncomprehension of the H or C of the H or C of the H or C and of theHabits of the H or C. and, given the and ability to ability to apply itof Mind and Given the opportunity, can apply it to to numerousFoundational opportunity, use it some, but not new/diverse Conceptssometimes consistently many, contexts (“HC”) uses the H or new/diverseUnable to Identifies the Can identify the Can identify Can identify theidentify the key elements of key elements of the key key elements of keyelements an argument, an argument, elements of an an argument of an butdoes not and can critique argument, and and effectively argument.critique them. them with some can critique critique them ineffectiveness, them with both, familiar but ONLY in some and novelexamples effectiveness, contexts provided in in both *OR* only novelclass. examples contexts. provided in class and in similar examples

FIG. 49 illustrates an new pop-up window 4900 generated in response toselection of an option from the pop-up window 4800 (“(I) ThinkingCritically” in the example) and selecting the “Next” button. In theillustrated pop-up window, additional sub-options associated with theinitial selection are illustrated including “(IA) Evaluating Claims,”“(IB) Analyzing and Organizing,” and “(IC) Making Decisions.” Thus, eachselection within a sequence of pop-up menus may generate a new set ofoptions, until the bottom of the selection hierarchy has been reached.In this manner, various evaluation criteria may be specified for thedebate by the instructor or other moderator. Responses may then besubmitted with respect to the evaluation criteria during or after thedebate, using an evaluation rubric such as shown in Table 1 for eachevaluation criteria.

FIG. 50 illustrates an example user interface for a debate. In thisillustrated example, the instructor has chosen to select twoparticipants to debate a topic and a first region 5000 is reserved forthe video image of the first debater and a second region 5001 isreserved for the video image of the second debater. In one embodiment,the instructor may manually select the participants by clicking anddragging the thumbnail video images 4601 corresponding to theparticipants into a corresponding one of the two regions 5000-5001. Asindicated by the arrows, a first video thumbnail of a first participantis clicked and dragged into region 5000 and a second video thumbnail ofa second participant is clicked and dragged into region 5001. In oneembodiment, upon selecting the two participants from the instructor'sclient, the state synchronization service 120 implements itspublish/subscribe mechanism (described in detail above) to ensure thatall clients are configured in a consistent manner (e.g., with video ofthe two debaters positioned in the central speaker regions 5000-5001 ofthe display).

Returning to FIG. 45, in one embodiment, the discussion maintenancemodule 4504 maintains a discussion by engaging all web conferenceparticipants. Debaters actively participate in the discussion bypresenting input (e.g., verbal arguments) consistent with the positionthat they have been asked to take. All web conference participants stayactively engaged in the debate by providing an evaluation of eachdebater. The discussion maintenance module 4504 may actively requestfeedback from each participant as the debate progresses over time. Itmay request evaluations, for example, using the rubric in Table 1 tovote on each participant during and/or after the debate. A webconference leader may specify a time period in which an evaluation isrequired. For example, the evaluation may be limited to certain specifictime windows (e.g., once every five minutes, once in the first 5 minutesand again at the last 5 minutes, etc). In one embodiment, the discussionmaintenance module 4504 aggregates the evaluation data provided by eachof the participants, combines the data (e.g., taking an average) andgenerates an evaluation in real-time. The evaluation may then beprovided to the discussion participants (the debaters and/or observers)in real-time.

FIG. 50 illustrates an example user interface for evaluating adiscussion participant. In the illustrated example, an observer mayevaluate a discussion participant by using a menu with a button systemdisplayed on their user interface. The options of the buttons within themenu correspond on evaluation criteria (e.g., an evaluation rubric). Adiscussion participant may be evaluated by using other systems. Forexample, evaluations may be provided by voting on a secondary device orwith keyboard shortcuts or with hand gestures (e.g., using LEAP Motioncontroller). Also noted in the user interface is information displayedalong a bottom edge corresponding to the debate topic.

FIG. 51 illustrates an exemplary graphical user interface in which thetwo debaters have been selected, thereby causing video images of the twodebaters to be displayed within regions 5000-5001. A first voting button5100 is displayed under the first region 5000 to allow participants tovote for the first debater and a second voting button 5101 is providedbeneath the second region to allow participants to vote for the seconddebater.

As illustrated in FIG. 52, in one embodiment, when a participant selectsa voting button 5100, a set of selectable evaluation options isgenerated within a pop-up menu 5200. In the illustrated example, theuser's performance may be rated from 0 (unsatisfactory) to 4(excellent). However, various other review criteria may be provided. Aseach user selects an option from the menu, scores are compiled for eachuser. As mentioned, the votes can be presented to the participants inreal-time or can be concealed from the participants (e.g., until thedebate is over). The votes may be further presented to observers and theweb conference leader. The web conference leader may specify thevisibility of an evaluation while preventing viewing by others. In theexample user interface shown in FIG. 53, the score 5300-5301 for eachdebater is visually display along with graphs 5310-5311 showing thevariability of the scores over time (i.e., as the debate progresses). Inone embodiment, the scores 5300-5301 comprise an average value of allvotes submitted for each debater.

A web conference leader/moderator may control the starting and theending time of a discussion as well as enable or disable evaluationswithin designated time periods. Additionally, each debater may be givena predetermined amount of time to make statements in different segments.For example, 5 minutes may be given for an opening statement, 10 minutesfor analysis and 5 minutes for a conclusion. As illustrated in FIG. 54,the time period and/or a timer 5400-5401 may be made visible to the webconference participants to indicate how much time is available for thecurrent segment.

FIG. 55 illustrates an example user interface with a termination button5500 for terminating a discussion. In one embodiment, the terminationbutton 5500 is only made available to the leader/moderator. In oneembodiment, during the course of the debate, the moderator may generateflags or bookmarks at certain points in time using user interfacecontrols so that the flagged or bookmarked portion of the debate can bereviewed at a later time (e.g., using the event filtering techniquesdescribed above, where the instructor can filter using bookmarks). Inaddition to bookmarking, one embodiment of the invention provides theinstructor the ability to provide comments at different stages of thedebate, which can later be reviewed by the debaters, the instructor,and/or other participants. Of course, the instructor or other webconference leader may choose the participants (debaters and/orobservers) to be provided with access to bookmarks and/or comments. Inone embodiment, the portions of a discussion with the greatest and leastvariability (i.e., agreement in votes) or disparity of votes (i.e.,differences in ratings for the debaters) may be highlighted andsubsequently reviewed. This disparity can be either among theevaluations of a single debater or between the overall evaluation ofeach debater, indicating junctures where one is clearly outperformingthe other. A participant may be required to evaluate regularly during adiscussion such as once every predetermined time interval (e.g., every 5minutes) or at the end of each segment (e.g., following each of theopening statement, analysis, and conclusion). As such, variability ordisparity can be computed both within each participant and acrossparticipants.

As described above, virtual conference sessions may be recorded. If thevirtual conference includes a debate, the discussion may be organizedaccording to each discussion participant's performance and evaluation.For example, clips of a specific duration (e.g., 10 seconds) with one ormore evaluation characteristics may be identified and ordered by theevaluation characteristics. A sequence of more-to-less highly variableevaluations or of greatest-to-least disparity in overall votes (i.e., awinner most clearly indicated to least clearly indicated) may bedisplayed.

A web conference leader may also set up a discussion before or duringthe class or web conference. Discussion participants may be selected bythe web conference leader or by the system. The system may selectdiscussion participants randomly or based on answers to a previous pollor clicker-style question, the split between performances of members ofbreakout, groups, or other information (e.g., nationality) related tothe participants. As mentioned, in one embodiment, the techniquesdescribed above with respect to the decision support module 3000 may beused to select the debaters (e.g., selecting debaters who would mostbenefit from the debate). Given a structured rubric and an archive ofrecorded discussion and the associated evaluation, participants cansearch for and review discussion participants who demonstrated good orbad skills defined in the evaluation rubric to serve as examples. Thisis done, in one embodiment, by generating a filtered event list 3502 byselecting certain participants and event types as described above.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules (e.g., code embodied on a machine-readable medium or ina transmission signal) or hardware modules.

A hardware module is tangible unit capable of performing certainoperations and may be configured or arranged in a certain manner. Inexample embodiments, one or more computer systems (e.g., a standalone,client or server computer system) or one or more hardware modules of acomputer system (e.g., a processor or a group of processors) may beconfigured by software (e.g., an application or application portion) asa hardware module that operates to perform certain operations asdescribed herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a field programmable gatearray (FPGA) or an application-specific integrated circuit (ASIC)) toperform certain operations.

A hardware module may also comprise programmable logic or circuitry(e.g., as encompassed within a general-purpose processor or otherprogrammable processor) that is temporarily configured by software toperform certain operations. It will be appreciated that the decision toimplement a hardware module mechanically, in dedicated and permanentlyconfigured circuitry, or in temporarily configured circuitry (e.g.,configured by software) may be driven by cost and time considerations.

The various operations of example methods described herein may beperformed, at least partially, by one or more processors, e.g.,processor 102, that are temporarily configured (e.g., by software) orpermanently configured to perform the relevant operations. Whethertemporarily or permanently configured, such processors may constituteprocessor-implemented modules that operate to perform one or moreoperations or functions. The modules referred to herein may, in someexample embodiments, comprise processor-implemented modules.

The one or more processors may also operate to support performance ofthe relevant operations in a “cloud computing” environment or as a“software as a service” (SaaS). For example, at least some of theoperations may be performed by a group of computers (as examples ofmachines including processors), these operations being accessible via anetwork (e.g., the Internet) and via one or more appropriate interfaces(e.g., application program interfaces (APis).)

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but deployed across a number of machines. In some example embodiments,the one or more processors or processor-implemented modules may belocated in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Some portions of this specification are presented in terms of algorithmsor symbolic representations of operations on data stored as bits orbinary digital signals within a machine memory (e.g., a computermemory). These algorithms or symbolic representations are examples oftechniques used by those of ordinary skill in the data processing artsto convey the substance of their work to others skilled in the art. Asused herein, an “algorithm” is a self-consistent sequence of operationsor similar processing leading to a desired result. In this context,algorithms and operations involve physical manipulation of physicalquantities. Typically, but not necessarily, such quantities may take theform of electrical, magnetic, or optical signals capable of beingstored, accessed, transferred, combined, compared, or otherwisemanipulated by a machine. It is convenient at times, principally forreasons of common usage, to refer to such signals using words such as“data,” “content,” “bits,” “values,” “elements,” “symbols,”“characters,” “terms,” “numbers,” “numerals,” or the like. These words,however, are merely convenient labels and are to be associated withappropriate physical quantities.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or a combination thereof), registers, or othermachine components that receive, store, transmit, or displayinformation.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or Bis satisfied by any one of the following: A is true(or present) and Bis false (or not present), A is false (or not present)and Bis true (or present), and both A and Bare true (or present).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the invention. Thisdescription should be read to include one or at least one and thesingular also includes the plural unless it is obvious that it is meantotherwise.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs for asystem and a process for providing a web conference leader withinformation about a set of participants who may benefit the most fromactive participation through the disclosed principles herein. Thus,while particular embodiments and applications have been illustrated anddescribed, it is to be understood that the disclosed embodiments are notlimited to the precise construction and components disclosed herein.

Various modifications, changes and variations, which will be apparent tothose skilled in the art, may be made in the arrangement, operation anddetails of the method and apparatus disclosed herein without departingfrom the spirit and scope defined in the appended claims.

Embodiments of the invention may include various steps, which have beendescribed above. The steps may be embodied in machine-executableinstructions which may be used to cause a general-purpose orspecial-purpose processor to perform the steps. Alternatively, thesesteps may be performed by specific hardware components that containhardwired logic for performing the steps, or by any combination ofprogrammed computer components and custom hardware components.

As described herein, instructions may refer to specific configurationsof hardware such as application specific integrated circuits (ASICs)configured to perform certain operations or having a predeterminedfunctionality or software instructions stored in memory embodied in anon-transitory computer readable medium. Thus, the techniques shown inthe figures can be implemented using code and data stored and executedon one or more electronic devices (e.g., an end station, a networkelement, etc.). Such electronic devices store and communicate(internally and/or with other electronic devices over a network) codeand data using computer machine-readable media, such as non-transitorycomputer machine-readable storage media (e.g., magnetic disks; opticaldisks; random access memory; read only memory; flash memory devices;phase-change memory) and transitory computer machine-readablecommunication media (e.g., electrical, optical, acoustical or other formof propagated signals—such as carrier waves, infrared signals, digitalsignals, etc.). In addition, such electronic devices typically include aset of one or more processors coupled to one or more other components,such as one or more storage devices (non-transitory machine-readablestorage media), user input/output devices (e.g., a keyboard, atouchscreen, and/or a display), and network connections. The coupling ofthe set of processors and other components is typically through one ormore busses and bridges (also termed as bus controllers). The storagedevice and signals carrying the network traffic respectively representone or more machine-readable storage media and machine-readablecommunication media. Thus, the storage device of a given electronicdevice typically stores code and/or data for execution on the set of oneor more processors of that electronic device. Of course, one or moreparts of an embodiment of the invention may be implemented usingdifferent combinations of software, firmware, and/or hardware.

Throughout this detailed description, for the purposes of explanation,numerous specific details were set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however, toone skilled in the art that the invention may be practiced without someof these specific details. In certain instances, well known structuresand functions were not described in elaborate detail in order to avoidobscuring the subject matter of the present invention. Accordingly, thescope and spirit of the invention should be judged in terms of theclaims which follow.

What is claimed is:
 1. A system for tracking events and providingfeedback in a virtual conference, the system comprising: a plurality ofclients operated by participants and at least one moderator of thevirtual conference; a virtual conferencing service to establish audioand/or video connections between the plurality of clients during thevirtual conference; a contribution identification module to identifyevents related to actions of participants during the virtual conference;an event log to store the events identified by the contributionidentification module; an event filter to search for specific types ofevents within the event log based on input from a moderator and/or theparticipants when reviewing a recording of the virtual conference, theevent filter comprising options selectable by the moderator and/or theparticipants for filtering events by participant and by different eventtypes, wherein the event filter is to generate a filtered set of eventsbased on the input from the moderator and/or participants; an event listto visually display the filtered set of events within a graphical userinterface (GUI); and a contribution assessment module to provide anoption to enter feedback related to one or more events selected withinthe event list.
 2. The system as in claim 1, further comprising: a mediaplayer to play back recorded portions of video and/or audio from thevirtual conference corresponding to an event selected from within theevent list.
 3. The system as in claim 2, wherein the media playerincludes a graphical element to allow the user to jump ahead or jumpback to different portions of the video and/or audio, wherein uponjumping ahead or back, an event is highlighted from within the eventlist corresponding to the point to which the jump was performed withinthe video and/or audio.
 4. The system as in claim 1, wherein thedifferent types of events include speaking events comprising periodsduring which one or more of the participants speak during the virtualconference, typed comments submitted by participants during the virtualconference, and bookmark events indicating points in time at which themoderator entered bookmarks during the virtual conference.
 5. The systemas in claim 1, wherein the event filter further includes options forperforming additional filtering within a particular type of event. 6.The system as in claim 5, wherein the event filter provides options forfiltering speaking events based on the duration of the speaking eventsand further provides options for filtering typed comments based onlength of the typed comments.
 7. The system as in claim 1, wherein thecontribution assessment module is to generate a feedback region with theGUI into which a moderator and/or participant may enter the feedback. 8.The system as in claim 7, wherein the contribution assessment module isto provide a list of options from which the moderator and/or participantmay select to provide the feedback for each selected event.
 9. Thesystem as in claim 8, wherein the contribution assessment module is toprovide a field into which moderator and/or participant may providecomments related to each selected event.
 10. The system as in claim 9,wherein a participant is provided with access to the comments and otherfeedback related to an event by selecting the event from within theevent list.
 11. The system as in claim 10, wherein the event filterfurther includes an option to filter events based on whether the eventshave been assessed.
 12. A method for tracking events and providingfeedback in a virtual conference, the method comprising: establishingaudio and/or video connections between a plurality of clients with avirtual conferencing service; rendering a virtual conferencing graphicaluser interface (GUI) on each client; identifying events related toactions of participants during the virtual conference; storing theevents within an event log; providing options for searching for specifictypes of events within the event log based on input from a moderatorand/or participants when reviewing a recording of the virtualconference, the options being selectable by the moderator and/or theparticipants for filtering events by participant and by different eventtypes; generating a filtered set of events based on the input from themoderator and/or participants; visually displaying the filtered set ofevents within an event list of a graphical user interface (GUI); andproviding an option to enter feedback related to one or more eventsselected within the event list.
 13. The method as in claim 12, furthercomprising: playing back recorded portions of video and/or audio fromthe virtual conference within a media player, the recorded portions ofvideo and/or audio corresponding to an event selected from within theevent list.
 14. The method as in claim 13, wherein the media playerincludes a graphical element to allow the user to jump ahead or jumpback to different portions of the video and/or audio, wherein uponjumping ahead or back, an event is highlighted from within the eventlist corresponding to the point to which the jump was performed withinthe video and/or audio.
 15. The method as in claim 12, wherein thedifferent types of events include speaking events comprising periodsduring which one or more of the participants speak during the virtualconference, typed comments submitted by participants during the virtualconference, and bookmark events indicating points in time at which themoderator entered bookmarks during the virtual conference.
 16. Themethod as in claim 12, further comprising: providing options forperforming additional filtering within a particular type of event. 17.The method as in claim 16, wherein the options include options forfiltering speaking events based on the duration of the speaking eventsand further include options for filtering typed comments based on alength of the typed comments.